Questions tagged [wget]
GNU Wget (or just wget) is a command line-based program for downloading content from the Internet using HTTP, HTTPS and FTP.
500
questions
285
votes
6
answers
856k
views
How do I provide a username and password to wget?
I've tried to connect to a server via wget:
wget http://<user>:<pass>@serveradress
But wget responds: invalid port
I know that the server accepts incoming traffic at port 80.
How can I ...
- 3,819
163
votes
8
answers
233k
views
Download files from a list
How can I download files (that are listed in a text file) using wget or some other automatic way?
Sample file list:
www.example.com/1.pdf
www.example.com/2.pdf
www.example.com/3.pdf
- 2,305
73
votes
2
answers
163k
views
How to make wget faster or multithreading?
I've just upgraded my computer hardware(cpu + motherboard + graphic card + memory + hard disk), so that install a new OS is needed. I tried to download debian-6.0.6-amd64-netinst.iso with wget command ...
- 1,312
66
votes
9
answers
148k
views
How do I use wget/curl to download from a site I am logged into?
Some parts of wikipedia appear differently when you're logged in. I would like to wget user pages so they would appear as if I was logged in.
Is there a way I can wget user pages like this
http://...
- 10.9k
56
votes
3
answers
31k
views
Force wget to use actual filename
When using wget in a script to download some files from Google Docs, the name of the file is not preserved. For example:
wget 'http://spreadsheets.google.com/pub?key=pyj6tScZqmEfbZyl0qjbiRQ&...
- 911
54
votes
5
answers
150k
views
Download a whole website with wget (or other) including all its downloadable content
I'm trying to download winamp's website in case they shut it down. I need to download literally everything.
I tried once with wget and I managed to download the website itself, but when I try to ...
- 9,790
47
votes
4
answers
117k
views
How to get WGET to download exact same web page html as browser
Using a web browser (IE or Chrome) I can save a web page (.html) with Ctl-S, inspect it with any text editor, and see data in a table format. One of those numbers I want to extract, but for many, many ...
- 471
45
votes
4
answers
40k
views
What is the difference between curl and wget? [closed]
Both of these curl and wget are use for downloading files, etc. What are the notable differences one might take in account in order chose one over the other?
- 4,040
37
votes
6
answers
121k
views
How to verify the SSL fingerprint by command line? (wget, curl, ...)
Using a command line website downloader, such as wget, curl or any other one... In a script...
I have the SHA-1 and the SHA-256 certficate fingerprint of a website. Due to security concerns (1) (2), ...
34
votes
3
answers
416k
views
bash: syntax error near unexpected token `('
I am trying to download flareget download manager via wget
I get error
wget http://www.flareget.com/files/flareget/debs/amd64/flareget_2.3-24_amd64(stable)_deb.tar.gz
bash: syntax error near ...
- 4,994
33
votes
2
answers
158k
views
gzip: stdin: not in gzip format
When I run:
tar -zxvf john-1.7.0.2.tar.gz
I get this:
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
It should start unpacking.
Some ...
- 431
31
votes
5
answers
249k
views
How do I force wget to use a proxy server without modifying system files? [duplicate]
I'd like to know an equivalent to wget enabling me to set the proxy while calling the command (like wget --proxy=$http_proxy).
Any ideas ?
- 415
30
votes
2
answers
71k
views
This command can only be used by root
I'm on Ubuntu 16 and I'd like to add a neo4j package. I tried in 3 ways all leading to the same error.
These are my attempts:
wget -O - http://debian.neo4j.org/neotechnology.gpg.key | apt-key add -
...
- 403
28
votes
1
answer
24k
views
`curl -O` stores an empty file though `wget` works well
I have tried to download a file from https://logz.io/sample-data.
curl -O https://logz.io/sample-data
However, it just returns an empty file named sample-data.
In contrast to curl, however, wget ...
- 415
26
votes
2
answers
11k
views
How to determine the HTTP status without downloading the complete page?
I want to know the HTTP Status of the websites using Ubuntu.
I have used curl and wget command for that purpose. But the problem is these commands download the complete website page and then search ...
- 1,548
25
votes
3
answers
151k
views
How can I see/stop current running crontab tasks?
How can I see crontab tasks which are in progress?
How can I stop crontab tasks which are in progress, manually?
Is [this] question about see the current running cron processes?
Yes
To kill a ...
- 2,325
25
votes
1
answer
83k
views
How to install 'wget' on Ubuntu 14.04?
I read the documentation of wget but did not understand how to download the package.Is there any way (command) to download the package using the terminal on Ubuntu inside a Docker container?
- 859
24
votes
3
answers
82k
views
Get wget output to a variable
How can I do it? I am trying it like this
RESULT=`wget $URL`
P.S. - Also any recommended sources for learning shell scripting?
- 3,214
24
votes
4
answers
76k
views
I used wget to download html files, where are the images in the file stored?
Firefox was loading very slow, so I decided to use wget to save HTML files.I used the following command,
wget http://textbook.s-anand.net/ncert/class-xii/chemistry/hello-this-first-chapter
The files ...
- 9,631
24
votes
1
answer
19k
views
How can I download an HTML webpage including JavaScript-generated content from the terminal?
On Google Chrome, when we go to the development mode, right-click an HTML element → Copy → Copy element, we can copy the HTML content of a webpage. Below is an example of the procedure I've described:
...
- 493
24
votes
1
answer
20k
views
How do I restart a wget download?
I use the wget application. One file has stopped downloading. How can I stop the download and start the download again?
Here's the log for wget
$ tail -f wget-log
287350K .......... .......... .........
- 313
23
votes
4
answers
124k
views
How to download a GitHub repo as .zip using command line
I am trying to download a .zip file from GitHub using the command line in Ubuntu. I am using wget command for it on a remote Ubuntu system.
I run wget <link> where <link> is the address ...
- 333
22
votes
1
answer
7k
views
How can I create a wget GUI with multiple progress bars?
I'm writing my own script to download multiple items (simultaneously, one item per command) using wget and I need a progress bar. I currently utilize zenity --progress for my progress bars, but the ...
- 2,238
21
votes
4
answers
53k
views
"/dev/fd/63 No such file or directory"
I've been trying to run a command in chroot (pretty new to this), and I get the following output.
root@hostname:~ # bash <(wget -q0 https://raw.githubusercontent.com/ubports/unity8-desktop-install-...
- 313
20
votes
2
answers
412k
views
Installing apt-get
The apt-get command and the sudo command does not work for me. which apt-get returns nothing. I checked by bin folder to make sure, and there is no aptget. I however have wget. Can I install apt-get ...
- 303
19
votes
3
answers
52k
views
How to download and install Heroku?
I downloaded Heroku from Heroku toolbelt using
wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
in the terminal, which they recommended on their site. My Ubuntu (14.04.1) didn't ...
- 191
18
votes
2
answers
109k
views
Getting "Scheme missing" error with wget
I wrote a little script that grabs a random wallpaper from the Desktoppr API and changes my desktop wallpaper to it:
#!/bin/bash
url=$(curl 'https://api.desktoppr.co/1/wallpapers?page='$(shuf -i 1-...
- 333
18
votes
2
answers
28k
views
How to download .mp3 files whole site?
I want to download all mp3 files in the website (even in home page and sublinks).
For example I want to download all .mp3 links from the "http://example.org/musics/" and all sub-links.
How can I do ...
- 361
16
votes
6
answers
12k
views
Why would I use Wget instead of a browser? [closed]
In what case should I prefer to use Wget rather than a browser?
I heard that Richard Stallman uses it instead of a browser for some anonymity reasons. And what does the server see when you get its ...
- 2,439
16
votes
4
answers
26k
views
Can I stop wget creating duplicates?
If I run wget twice, it doesn't recognise that it has already downloaded that file, and creates a new one. Is there any way to prevent it downloading the file again?
$ wget https://cdn.sstatic.net/...
- 20.9k
16
votes
2
answers
25k
views
Get all image files with wget
I am trying to download all the image file in a folder. I tried
wget -r https://example.com/
And
wget https://example.com/*.jpg
These didn't work. Can anyone guide?
- 297
16
votes
1
answer
23k
views
how to make wget retry download if speed goes below certain threshold?
I'm trying to download a large 2GB file with wget, but after a few minutes it seems to keep stalling.
I ^C it, and wget it again [with --continue option], and it starts downloading again.
Is there ...
- 9,993
16
votes
2
answers
18k
views
how to resume download in wget which is integrated with firefox via flashgot
I use wget as a downloader. It is an excellent, outstanding application. I integrated it with Firefox using Flashgot.
How can I resume a download which was stopped by a CTRL+C key press or a shutdown ...
- 831
14
votes
2
answers
106k
views
How to change wget default directory from 'Videos' to 'Downolads'?
I use the combination of Flashgot and wget for downloads. I reset my Firefox preferences due to some other problems. Then I reconfigured everything (including add-ons, cache, etc).
After that, when ...
- 831
14
votes
2
answers
19k
views
what does wget -qO- 127.0.0.1 mean?
I have tried googling for an answer for an hour now and still no luck. Does anyone know what this command mean?
wget -qO- 127.0.0.1
I know wget is the Linux "web get" utility, and 127.0.0.1 is ...
- 293
14
votes
4
answers
19k
views
How to download link with unicode using wget?
Sometimes a link has unicode characters in it, such as http://www.example.com/файл.zip
If you point your browser to it, it will properly prompt you to download the file as файл.zip. But if you try to ...
- 11.9k
14
votes
1
answer
22k
views
How to download a whole FTP site using command line?
I need to download an FTP site (a web site source code actually) containing a very high quantity of relatively small files (downloading it with FileZilla took more than a day, but I believe it could ...
- 57.1k
14
votes
2
answers
19k
views
How I can download PDFs of a website by using only the root domain name?
I am using this command:
wget -nd -e robots=off --wait 0.25 -r -A.pdf http://yourWebsite.net/
but I can't get PDFs from the website.
For example I have a root domain name:
www.example.com
and ...
- 145
13
votes
3
answers
3k
views
How can I gradually encrypt a file that is being downloaded?
I used to download files and then use openssl to encrypt them, which is not enough for me. How can I encrypt the download gradually as the file is being downloaded?
I thought about using curl and an ...
- 2,378
13
votes
5
answers
21k
views
wget and encoded URL
I have a URL like this:
http://dl.minitoons.ir/longs/Khumba (2013) [EN] [BR-Rip 720p] - [www.minitoons.ir].rar
I want to download this URL using wget. If I pass it directly to wget, everything goes ...
- 365
13
votes
1
answer
31k
views
wget or curl gives 403:Forbidden while downloading file from OneDrive for business
I'm trying to download software for a client which resides in my bussiness OneDrive account. Through the webinterface of OneDrive I do:
Right click the file and select Share
List item
Change the ...
- 2,154
12
votes
6
answers
10k
views
How to retrieve tweets from command line?
Is there a simple way to retrieve someone's tweets from the command line?
The account would be public, and the command - or script, if necessary - would retrieve all or a specified number of most ...
- 11.9k
12
votes
2
answers
6k
views
wget script, but pipe it to bash only if SHA256 of script matches one-liner
wget http://example.com/install.sh -O - | bash runs the script automatically but due to possible (TLS) MITM attacks etc, this is not safe. Is is possible to construct a one-liner that downloads the ...
- 173
11
votes
3
answers
121k
views
Getting 'permission denied error' with Wget
I installed the LAMP stack on my Ubuntu 11.04 (Natty Narwhal) with
sudo apt-get install lamp-server^
Now I navigate to /var/www and use
wget http://ftp.drupal.org/files/projects/drupal-7.x-...
- 1,185
11
votes
2
answers
59k
views
Wget: download files to specific directory
I have been trying to get wget to download all files to a specific directory. I tried the -O option but I get
/home/user/xml/: Is a directory
This is what I have so far
wget -m --user=user --...
- 191
10
votes
6
answers
50k
views
How do I bypass the license page to download Oracle/Sun Java on a server with wget?
I'm trying to install Liferay on Ubuntu Server 12.04, which requires JRE and recommends version 6.
I can't find JRE6 in an authoritative repository, and I can't work out how to get past the licence ...
- 203
10
votes
1
answer
7k
views
Purpose of .wget-hsts file in Ubuntu
There is a file in my home folder named .wget-hsts. Does this file send my data to anywhere? The file content is as follows:
# HSTS 1.0 Known Hosts database for GNU Wget.
# Edit at your own risk.
# &...
- 203
10
votes
3
answers
2k
views
How do I create a CLI Web Spider that uses keywords and filters content?
I want to find my articles within the deprecated (obsolete) literature forum e-bane.net. Some of the forum modules are disabled, and I can't get a list of articles by their author. Also the site is ...
- 29.9k
9
votes
1
answer
9k
views
Why is wget still shipped as the default download manger, whereas there's richer apps [closed]
aria2c has invaluable features like:
Segmented downloading
Metalink Support
Download multiple files simultaneously
Multiple connections downloading
Torrent support with chunk checksum validation
...
- 387
9
votes
2
answers
20k
views
How to get link of file to download with wget
I am using Ubuntu 19.10. For automation purposes, I need to download this file with a script.
But I am not able to get the link.
How can I achieve this?
- 22.5k