2

I want to wget (or other download batch command) the latest file that is added to a large repository. The latest nightly build thru http. I could mirror all files, but the repository are huge so I want to be able to remove old files and only trigger when there is a new file.

2

A first simple approach would be to use the time stamping capacity of wget, with this, you could only download the new files from the repository. But it leads to make a mirror of the repository (but refreshed smartly with only the new files :) ).

  • Even though it mirrors the repository in local, but according to the wget documentation, But once you mirrow the repository in local after that it will only download the modified files. – Rajesh Pantula Jan 4 '12 at 14:29
  • I still have to keep the whole repository at my local site, less then a TB but large enought. – Anders Wallenquist Jan 8 '12 at 21:50
0

I would advise you use rsync for this task. You have several options on how to do this, there are scripts available here

https://wiki.ubuntu.com/Mirrors/Scripts

And there is a full tutorial here:

https://help.ubuntu.com/community/Rsyncmirror

And debmirror:

https://help.ubuntu.com/community/Rsyncmirror

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.