15

In what case should I prefer to use Wget rather than a browser?

I heard that Richard Stallman uses it instead of a browser for some anonymity reasons. And what does the server see when you get its files using Wget?

closed as too broad by Pilot6, Anwar, edwinksl, David Foerster, Eric Carvalho Sep 6 '16 at 15:20

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    If all I need is to download, say, a file, why would I waste resources opening an entire browser to do it? Where did you hear that Stallman uses wget for "some security reasons"? – edwinksl Sep 5 '16 at 1:48
  • 3
    There are at least four questions here: 1. what are the advantages of wget vs a browser; 2. how does wget present to a webserver; 3. does wget have any privacy benefits over a web browser; 4. does [Richard] Stallman use wget, and if so, is it for anonymity. – Ben Aveling Sep 5 '16 at 8:31
  • 4
    According to this How I do my computing page, Stallman uses something "much like wget": fetching web pages per email (but he does use graphical browsers to open the fetched pages, if needed); he also browses the Web via Tor, using a graphical browser, so fetching via email is not the only way. – unor Sep 5 '16 at 10:43
  • 3
    Please post the source of your hearing – Anwar Sep 5 '16 at 13:27
  • 3
    «And what server thinks/sees when you get its files using Wget?» it thinks “oh, that must be Stallman! I'll personalize the content to mess with him.” – JDługosz Sep 5 '16 at 17:16
25

Typically you would never use it "instead of a browser". Browsers render HTML, make links clickable (as opposed to having to copy the URL into another wget command manually), etc. There's literally no upside to using wget as a human. If you are concerned about privacy, there's a million ways to clean a browser up (or you could use a less featureful browser, like Lynx if you really wanna get barebones without destroying all semblance of human user interface).

Wget is primarily used when you want a quick, cheap, scriptable/command-line way of downloading files. So, for example, you can put wget in a script to download a web page that gets updated with new data frequently, which is something a browser can't really be used for. You can use wget's various options to crawl and automatically save a website, which most browsers can't do, at least not without extensions.

In short, browsers are applications for humans looking at the internet, wget is a tool for machines and power users moving data over HTTP. Very similar in what they do (pull files from websites) but entirely different in their use.

Regarding what servers "see" when you get things with wget: all HTTP clients (browsers, wget, curl, other similar applications) transmit what's called a "User Agent", which is just a string that describes the browser (or these days, describes what browser features it has). This can be used to show different content depending on the user's browser (i.e. Google tries not to advertise Chrome to people already using Chrome). Some fools try to block power user shenanigans by blocking wget's user agent string, but you can just fake a Chrome user agent string to get around that. More often it's simply used for statistics so you know how popular different browsers are so you know which ones to test with the most thoroughly.

If you use wget's crawling functions, the server will see many rapid requests in a mostly alphabetical order. It's a dead giveaway that you're scraping their site. It looks entirely different from the browsing of a user. With a human user making requests in a browser, every page request is followed by all the images on that page, and then there's some delay, and then there's a request for another random page (or possibly a string of pages with a clear purpose).

16

As others have mentioned, wget has the benefit of not being bundled with add-ons, cookies, and cache, which makes it potentially more stable and secure. But browsers and wget actually have very different normal uses.

wget is a command-line utility meant to retrieve content, not to present it. It can be used to retrieve and download anything through FTP, HTTP, and HTTPS, including any file types (HTML, images, binaries, etc).

For the server, the only difference it will see is a different user agent unless you use its --user-agent argument to specify a browser's. If you do, the server won't see any difference.

5

1) downloads initiated by a script rather than a human being

2) downloading whole sites (or fragments of sites) rather than separate pages. (Wget can automatically follow links.)

Wget has some command-line options to control what the server sees and can think, including arbitrary delays between requests to download pages. But if the site server has some anti-bot policy, you often waste a large amount of time and traffic before you get an acceptable result.

  • 3) Download initiated by a human when there is no browser easily accessible (e.g. over ssh) – Dezza Sep 6 '16 at 9:50
4

When using a browser anything within it is active, including undesirable addons and even malware, whereas wget is a simple command that downloads stuff and does absolutely nothing else.

The browser is not necessarily less secure but potentially so.

The servers "see" a download request, as usual.

4

The browser can record your activities in history and cache files.

wget has no memory retention.

That said a browser is more convenient and I don't care if the NSA figures out I downloaded 4.7.2-040702-generic last week. I use my browser for on-line banking and paypal which is arguably more crucial than the majority of files I can download with wget.

2

As a developer who uses remote Ubuntu installations, there is often very little point in installing the OS GUI. In this 'command line' environment, wget is an essential tool for downloading files from the web.

I'm not even sure which, if any, browsers can be run from terminal.

Not the answer you're looking for? Browse other questions tagged or ask your own question.