I'm using wget to download a list of files from a URL, here is my command:

wget --user='username' --password='mypassword' -i  url.txt

This downloads a list of .gz files

What I need to do is:

  1. Before downloading, delete all files in directory
  2. After downloading, unzip all files and delete zipped ones

Is there a command or list of commands which might accomplish this which I can insert into a cronjob? I need to run this once a day.



Let's try a one-liner :

rm -rfi * && wget -q -O - --user='username' --password='mypassword' -i  url.txt | wget - && find . -name "*.zip" -exec "unzip && rm" {} \;

Don't hesitate to edit/adapt : this could be really wrong as I'm not always very at ease with the - pseudo-descriptor and how each application handles it.

Some details

Clear current directory.

rm -rfi *

Get the list, but instead of sending the contents to a file, redirect it to stdout (using -q, -O and the pseudo-descriptor -).

wget -q -O - --user='username' --password='mypassword' -i  url.txt 

Download files names of which are being sent from the previous wget call. Again, - makes refers to the standard input, which is also the previous command's output.

wget -

Find all zip files in the directory, and execute unzip and rm on each of them.

find . -name "*.zip" -exec "unzip && rm" {} \;
  • Could you explain what this command does and how? – Seth Nov 27 '13 at 19:41
  • Sure. I tried my best. Edited. – John WH Smith Nov 27 '13 at 19:47
  • @JohnWHSmith you should probably add a caveat that rm -rf * will delete ALL files - including any url.txt file that is in the working directory - if the url.txt file is to be preserved it needs to be moved out of the directory or excluded from the rm glob – steeldriver Nov 27 '13 at 19:53
  • It may get more complicated than I thought in the end... Adding -i to the rm command keeping it "simple". – John WH Smith Nov 27 '13 at 20:07

This is more of a *nix question, than Ubuntu-centric.

In general you'd write a script (using the bash language, or perhaps Python) that handles the task you describe. One line of the script would be your command, above... the rest would handle the inspection of the directory, deletion, unzipping etc.

Then you'd just use cron (google up crontab) to set the script to run at specific times.

  • Actually I've managed to solve the issue, by using ; to separate all my commands to run one after the other. The only question I still can't solve is how to unzip all of the .gz files in the directory, using the normal 'unzip' command doesn't work – jc.yin Nov 27 '13 at 19:24
  • @jc.yin Why are you using unzip to unzip a tar.gz file? You should use tar. MikeW: This isn't really an answer in it's current form. Could you expand on how to do what you're saying? Also, non "Ubuntu-centric" questions are OK here. – Seth Nov 27 '13 at 19:41
  • you could use shell for loop. for z in *.zip; do unzip $z; done – souravc Nov 27 '13 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.