Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

Some parts of wikipedia appear differently when you're logged in. I would like to wget user pages so they would appear as if I was logged in.

Is there a way I can wget user pages like this

http://en.wikipedia.org/wiki/User:A

this is the login page:

http://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Login&campaign=ACP3
share|improve this question
    
See also this article: [How to download this webpage with wget?][1] [1]: superuser.com/questions/214846/… –  kenorb Feb 20 '13 at 18:34
add comment

4 Answers

up vote 16 down vote accepted

The easy way: login with your browser,and give the cookies to wget

Easiest method: in general, you need to provide wget or curl with the (logged-in) cookies from a particular website for them to fetch pages as if you were logged in.

If you are using Firefox, it's easy to do via the Export Cookies add-on. Install the add-on, and:

  1. Go to Tools...Export Cookies, and save the cookies.txt file (you can change the filename/destination).
  2. Open up a terminal, and use wget with the --load-cookies=FILENAME option, e.g.

    wget --load-cookies=cookies.txt http://en.wikipedia.org/wiki/User:A
    
    • For curl, it's curl --cookie cookies.txt ...

(I will try to update this answer for Chrome/Chromium users)

The hard way: use curl (preferably) or wget to manage the entire session

  • A detailed how-to is beyond the scope of this answer, but you use curl with the --cookie-jar or wget with the --save-cookies --keep-session-cookiesoptions, along with the HTTP/S PUT method to log in to a site, save the login cookies, and then use them to simulate a browser.
  • Needless to say, this requires going through the HTML source for the login page (get input field names, etc.), and is often difficult to get to work for sites using anything beyond simple login/password authentication.
  • Tip: if you go this route, it is often much simpler to deal with the mobile version of a website (if available), at least for the authentication step.
share|improve this answer
add comment

Try something like:

wget --keep-session-cookies --save-cookies cookies.txt --post-data 'user=goyamy&passwrd=mypassword' http://forum.ubuntu-it.org/

See also this link:

How to download this webpage with wget?

share|improve this answer
add comment

For more complicated website based logins you should also consider to use a Python script and some module which imitates a browser, like http://wwwsearch.sourceforge.net/mechanize/ instead of curl or wget.

This way session cookies are handled automatically, you can follow links and fill login forms, and so "script" yourself through the login process as if using your web browser.

share|improve this answer
    
Yeah, using mechanize can be really nice. Also (and originally) available in Perl, if that is more your cup of tea. –  andol Apr 3 '13 at 8:52
add comment

Have you tried this?

wget --user=username --password=password http://en.wikipedia.org/wiki/User:A
share|improve this answer
    
yes - unfortunately that didn't work =( –  user784637 Jul 9 '12 at 23:53
5  
Note: this syntax works for basic HTTP authentication only (where your browser pops up a special Username/Password dialog box), not for webpage-based authentication. –  izx Jul 10 '12 at 3:30
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.