Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

As said in wget man page:

to download a single page and all its requisites (even if they exist on separate websites), and make sure the lot displays properly locally, this author likes to use a few options in addition to -p:

wget -E -H -k -K -p http://mysite.com/directory

I understand, if I want to download mysite entirely, I have to add -r argument. But using both -r and -H options yields downloading all the website reachable from http://mysite.com/directory. Any idea?

share|improve this question
add comment

2 Answers

Its been a while since I used wget for this purpose:

I believe I had success with the - m flag.

wget -mk http://site.com/directory

This probably won't get everything - but it will get you close.

(Reference): This page

share|improve this answer
    
-m does not work when the page requisites is in a separate site. –  Ali Aug 2 '12 at 13:03
add comment

If you want to use wget, you can use the mirror setting to make an offline copy of a website, although some websites might prevent it with their robots.txt settings that stops automated spidering. I have always had a few problems with wget (see my other suggestion below), but the following command does work for many sites. However, be aware that the addition of the -H switch allows it to access all links that are on other sites and to save those also. This command switch can obviously be removed if it is not required.

 wget --wait 1 -x -H -mk http://site.to.mirror/

The command to wait allows some gaps between wget's requests so that the site is not overwhelmed, and the -x command switch specifies that the site's directory structure should be exactly mirrored in a folder in your home folder. The -m switch obviously stands for mirror mode, which allows wget to download recursively through the site; and the -k switch means that after download the files referenced will be those in your mirror directory in your home folder and not those back at the site itself.

After man wget, perhaps the best listing and detailed explanation of wget commands is here.

If wget is unsuccessful and you can't grab as much as you want, I should try the command line program httrack or its web interface, webhttrack, which are available in the repositories. There are a large amount of options for this program, but it is better for downloading whole websites or parts of websites than wget. Webhttrack gives you a wizard to follow for downloading a site (it opens in your browser) as the screenshot below shows.

Httrack

share|improve this answer
    
-x -mk does not work when the page requisites is in a separate site. –  Ali Aug 2 '12 at 13:05
    
I have edited my wget command by adding the -H switch which will fetch the external links. Try httrack if you can't get wget to do what you want. –  user76204 Aug 2 '12 at 13:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.