59

Some parts of wikipedia appear differently when you're logged in. I would like to wget user pages so they would appear as if I was logged in.

Is there a way I can wget user pages like this

http://en.wikipedia.org/wiki/User:A

this is the login page:

http://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Login&campaign=ACP3
2

9 Answers 9

65

###The easy way: login with your browser,and give the cookies to wget

Easiest method: in general, you need to provide wget or curl with the (logged-in) cookies from a particular website for them to fetch pages as if you were logged in.

If you are using Firefox, it's easy to do via the cookie.txt add-on. Install the add-on, and:

  1. Click on the plugin and save the cookies.txt file (you can change the filename/destination).

  2. Open up a terminal, and use wget with the --load-cookies=FILENAME option, e.g.

     wget --load-cookies=cookies.txt http://en.wikipedia.org/wiki/User:A
    
  • For curl, it's curl --cookie cookies.txt ...

(I will try to update this answer for Chrome/Chromium users)

###The hard way: use curl (preferably) or wget to manage the entire session

  • A detailed how-to is beyond the scope of this answer, but you use curl with the --cookie-jar or wget with the --save-cookies --keep-session-cookiesoptions, along with the HTTP/S PUT method to log in to a site, save the login cookies, and then use them to simulate a browser.
  • Needless to say, this requires going through the HTML source for the login page (get input field names, etc.), and is often difficult to get to work for sites using anything beyond simple login/password authentication.
  • Tip: if you go this route, it is often much simpler to deal with the mobile version of a website (if available), at least for the authentication step.
5
  • 1
    Nice tip. The code for the mobile site is often way cleaner. Jun 4, 2016 at 8:39
  • 4
    how does such a cookies.txt look like? Jun 17, 2016 at 15:38
  • 1
    This "Export Cookies" add-on created a cookies file with the cookies for every single site I've accessed in Firefox. So if you follow this answer you'll be presenting all of your cookies to whatever site you are hitting. I consider this a security flaw in this answer. You can easily work around this problem by using: grep mydomain cookies.txt > mydomain-cookies.txt Jun 15, 2017 at 1:35
  • 3
    For chrome: chrome.google.com/webstore/detail/cookiestxt/…
    – Timo
    Mar 14, 2018 at 14:14
  • You can use Cookie Manager add-on in Firefox to export only selected cookies. Apr 25, 2018 at 7:32
21

Another easy solution that worked for me without installing anything extra:

This will give you a command that you can paste directly into your shell, that has all your cookie credentials e.g.

curl 'https://mysite.test/my-secure-dir/picture1.jpg' \ 
-H 'User-Agent: Mozilla/5.0 ...' \
-H 'Cookie: SESSIONID=abcdef1234567890'

You can then modify the URL in the command to fetch whatever you want.

1
  • This is a fantastic tip!
    – studog
    Dec 1, 2020 at 17:21
4

With cURL is really easy to handle cookies in both ways.

curl www.target-url.com -c cookie.txt then will save a file named cookie.txt. But you need to log in, so need to use --data with arguments like: curl -X --data "var1=1&var2=2" www.target-url.com/login.php -c cookie.txt. Once you get loggued cookie you can send it with: curl www.target-url.com/?user-page.php -b cookie.txt

Just use -c (--cookie) or -b (--cookie-jar) to save and send.

Note1: Using cURL CLI is a lot of easier than PHP and maybe faster ;)

For save the final content you can easily add > filename.html to your cURL command then save full html code.

Note2 about "full": Yo cannot render javascript with cURL, just get the source code.

2
  • 1
    -X parameter stands for "do post", but using --data=xxx -X is implicit, so you can remove it surely.
    – m3nda
    May 18, 2014 at 19:17
  • -b is read cookie
    – Timo
    Mar 14, 2018 at 14:32
3

For those still interested in this questions, there's a very useful Chrome extension called CurlWGet that allows you to generate a wget / curl request with authentication measures, etc. with one click. To install this extension, follow the steps below:

  1. Install the extension from the Chrome Webstore.
  2. Go the web page that would you like to download.
  3. Start the download.
  4. The extension will generate a link for you.

Enjoy!

1
  • And that extension was recognized to contain malware, sadly :( Nov 23, 2020 at 9:41
3

have a look at cliget for Firefox.

When you're about to download, on the final download dialog you get the option to copy the download as curl command line to the clipboard.

2

The blog post Wget with Firefox Cookies shows how to access the sqlite data file in which Firefox stores its cookies. That way one doesn't need to manually export the cookies for use with wget. A comment suggests that it doesn't work with session cookies, but it worked fine for the sites I tried it with.

1

Have you tried this?

wget --user=username --password=password http://en.wikipedia.org/wiki/User:A
5
  • yes - unfortunately that didn't work =(
    – user784637
    Jul 9, 2012 at 23:53
  • 6
    Note: this syntax works for basic HTTP authentication only (where your browser pops up a special Username/Password dialog box), not for webpage-based authentication.
    – ish
    Jul 10, 2012 at 3:30
  • basic HTTP authentication url format is admin:admin@www.url.com but as @izx has say, basic HTTP is browser feature, not PHP programming nor code. If someday you need to access an htaccess protected url you can use that mode ;D
    – m3nda
    May 18, 2014 at 19:12
  • 1
    I didn't downvote, but this is not a good way to authenticate: if you, by accident make such scripts public, people can use your password. Nov 23, 2014 at 14:51
  • 1
    What's with the downvotes? May not be a catch all answer but still an answer.
    – Overcode
    Aug 6, 2015 at 17:49
1

Try something like:

wget --keep-session-cookies --save-cookies cookies.txt --post-data 'user=goyamy&passwrd=mypassword' http://forum.ubuntu-it.org/

See also this link:

How to download this webpage with wget?

2
  • Just like any output. Use > filename.html to save the output at current dir or put a complete filepath. This applies both Linux and Windows systems.
    – m3nda
    May 18, 2014 at 19:14
  • So, im seeing that wget has many features like cURL, or viceversa. It can do POST, so it can do login to websites. Interesting.
    – m3nda
    May 18, 2014 at 19:16
1

For more complicated website based logins you should also consider to use a Python script and some module which imitates a browser, like http://wwwsearch.sourceforge.net/mechanize/ instead of curl or wget.

This way session cookies are handled automatically, you can follow links and fill login forms, and so "script" yourself through the login process as if using your web browser.

1
  • Yeah, using mechanize can be really nice. Also (and originally) available in Perl, if that is more your cup of tea.
    – andol
    Apr 3, 2013 at 8:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.