When I was trying to mirror a website with Java files, XML files, ActionScript files, etc, wget only seems to download the commonly known files such as xml, txt, etc.

It seems to not be downloading the .java, .as files.

Here is my command:

wget --no-parent --user=USERNAME--password=PASSWD-e robots=off --header "CUSTOM HEADERS" "http://URL" -r --timeout=25

There doesn't seem to be anything wrong with it to me. Once the download is finished, none of the .java or .as files are there.

Also is there a way to skip over every directory that has a string in it? For example, if a directory is called "backup3r98238943r3jrefi", can I skip it for having the word "backup" in it? I want to skip any directory located in any dir/subdir on the site with a keyword in it.

  • Is this in reference to a specific site? There might be a level of protection on the site to prevent this. may be relevant: stackoverflow.com/a/11786158/1487176 – Mateo Nov 8 '14 at 2:37
  • and maybe this: stackoverflow.com/questions/273743/… – Mateo Nov 8 '14 at 2:41
  • You some to have some odd spacing in your command. Is that deliberate or an copy-paste error? – muru Nov 8 '14 at 2:58
  • Probably just an error from when I was removing passwords and unnecessary headers. – user3924824 Nov 8 '14 at 3:07

You can use --reject or --exclude-directories to skip things:

wget ... --reject '*backup*' ...
wget ... --exclude-directories '*backup*' ...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.