1

Suppose I have a certain pages, e.g. www.abc.def/ghi/a.html, is there any command option to download all the pages under www.abc.def./ghi (i.e. www.abc.def/ghi/a.html, www.abc.def/ghi/b.html, ...)?

I've installed httrack and webhttrack but I don't know how to correctly use them.

  • 1
    wget -r www.abc.def./ghi/ – blade19899 Nov 6 '14 at 13:25
  • wow, the famous -r! Btw, does it try to download ALL the links, even that under different domains? i.e. if the web pages contains link to facebook pages and so on, it follows the links and try to download these external pages also? – tigerjack89 Nov 6 '14 at 13:31
  • I never experienced that! - although that can be set as an option. Dunno which one tho. – blade19899 Nov 6 '14 at 13:32
  • well, as expected, it tried to download everything, even if it were only pages internal at the given domain (i.e. it tried to download www.abc.def/*). Sorry @blade19899, the solution doesn't work :( – tigerjack89 Nov 6 '14 at 14:38
  • 1
    possible Unix.stackexchange question there – αғsнιη Nov 6 '14 at 15:14
3

Try wget's recursive download with restrictions:

wget -r --include-directories=ghi --no-parent www.abc.def
  • well, actually it works well, but at the same time it downloaded also stuffs like .pdfs or the robot.txt. Also, I like more the httrack options mainly because, if I try to download again the site at later point, it gives the opportunity to download only the updated stuffs (don't know if wget can have similar options) – tigerjack89 Nov 6 '14 at 15:13
  • For people who like online manuals: manpages.ubuntu.com/manpages/hardy/man1/wget.1.html – Bulrush Nov 6 '14 at 18:25
  • @Bulrush if you use Ubuntu's manpages, please either link to a current one (say trusty and not something antique like hardy) or use the short url if it is unambiguous (like manpages.ubuntu.com/wget). – muru Nov 6 '14 at 21:26
  • @muro, Thanks for the note, I didn't notice the manpage version. Will do. – Bulrush Nov 7 '14 at 12:22
3

Found a solution with httrack and is the simplest one.

httrack http://www.abc.def/ghi/ -O <output-folder>

the key is that, by default, httrack use the -D parameter:

can only go down into subdirs

You can find more options typing httrack --help. Here is an useful link

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.