1

When I was trying to mirror a website with Java files, XML files, ActionScript files, etc, wget only seems to download the commonly known files such as xml, txt, etc.

It seems to not be downloading the .java, .as files.

Here is my command:

wget --no-parent --user=USERNAME--password=PASSWD-e robots=off --header "CUSTOM HEADERS" "http://URL" -r --timeout=25

There doesn't seem to be anything wrong with it to me. Once the download is finished, none of the .java or .as files are there.

Also is there a way to skip over every directory that has a string in it? For example, if a directory is called "backup3r98238943r3jrefi", can I skip it for having the word "backup" in it? I want to skip any directory located in any dir/subdir on the site with a keyword in it.

4
  • Is this in reference to a specific site? There might be a level of protection on the site to prevent this. may be relevant: stackoverflow.com/a/11786158/1487176
    – Mateo
    Nov 8, 2014 at 2:37
  • and maybe this: stackoverflow.com/questions/273743/…
    – Mateo
    Nov 8, 2014 at 2:41
  • You some to have some odd spacing in your command. Is that deliberate or an copy-paste error?
    – muru
    Nov 8, 2014 at 2:58
  • Probably just an error from when I was removing passwords and unnecessary headers. Nov 8, 2014 at 3:07

2 Answers 2

1

You can use --reject or --exclude-directories to skip things:

wget ... --reject '*backup*' ...
wget ... --exclude-directories '*backup*' ...
0

This totally depends on the directory structure of the website. In most cases unless you are going to pirate the site—don't do that, mkay?—you will only be able to obtain the compiled output of any java code which is likely present on the server.

You want to mirror the entire site, so instead of using -r use -w:

wget -mpEk --exclude-directories '*backup*' --reject '*backup*' --no-parent --robots=off <FTP USER INFO> <URL>

When entering the URL, use the bottom most directory, if you can't get it to work below public_html level, try removing --no-parent, depending on how your .htaccess is setup and directory structure it may be that your files are outside of what you're scraping, in which case since you have FTP credentials, perhaps using the sftp command would be more of what you're looking for.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .