60

I've just upgraded my computer hardware(cpu + motherboard + graphic card + memory + hard disk), so that install a new OS is needed. I tried to download debian-6.0.6-amd64-netinst.iso with wget command but the speed is so slow that I could not bear. 4Kb/s ~ 17 Kb/s, slow like a running turtle, or even more slower if I use Chrome.

I've read the help information of wget, it seems like there are no options could make it more faster.

Is there anyway to make wget faster? Or is it possible to make it multi-threading download?

PS: my bandwidth is 4M. I use this command:

wget -c url  http://hammurabi.acc.umu.se/debian-cd/6.0.6/amd64/iso-cd/debian-6.0.6-amd64-netinst.iso
5
  • 1
    wget just uses your connection. So if its slow, thats your connection with the server. Maybe you are slow, maybe the server is. btw 4mbit = 0.5mb/s, not to mention loss etc
    – Dr_Bunsen
    Nov 7, 2012 at 10:03
  • @Dr_Bunsen thank you for your advice, I tried the command that @Gufran suggested: axel, compared width wget, axel is faster than ever. I think in most situations, the bottleneck of my downloading speed are 1. something occupied the bandwidth(as you said: I'm slow). 2. single-threading. 3. the server is slow. But i have nothing to do with the 1&3 point.
    – Teifi
    Nov 8, 2012 at 2:39
  • @Teifi One possibility if axel multi-thread perform better than wget when using the same remote server, latency between your box and the remote server is very high. Check your ping to remote server.
    – John Siu
    Jan 8, 2013 at 4:20
  • Have you tried HTTrack? httrack.com/page/1/en/index.html
    – amanthethy
    Aug 7, 2014 at 3:23
  • See stackoverflow.com/questions/3430810/…
    – rogerdpack
    Dec 25, 2021 at 20:29

2 Answers 2

81

Why not try axel? It is a fully fledged Command line based Downloader.

Install axel and spawn download by

axel -a -n [Num_of_Thread] link1 link2 link3 ...

where '[Num_of_Thread]' is the number of parallel connections to create for each link you want to download.

-a just show an improved progress bar.

Unlike many other download managers, Axel downloads all the data directly to the destination file, using one single thread. This saves some time at the end because the program doesn't have to concatenate all the downloaded parts.

4
  • I am wondering if there is a way, maybe via alias and wrapper to use axel when is available instead of curl or wget, unless there is more than one parameter to the command line.
    – sorin
    Mar 6, 2015 at 9:23
  • 2
    I think this proposal is not sufficient for the download of one file. Please, correct me if necessary. My attempt here askubuntu.com/q/813483/25388 Aug 17, 2016 at 18:26
  • Thank you SO MUCH for this :)
    – rjurney
    Feb 23, 2020 at 23:00
  • Re: "I think this proposal is not sufficient for the download of one file". It's actually the opposite, axel is designed to multi-thread download of a single file, especially where multiple source links exist.
    – Cieniek
    May 18 at 10:25
75

I tried axel upon Gufran's recommendation but it hugely disappointed me. My goal was to find a CLI replacement for DownThemAll because it hogs the CPU and hard disc and slows the entire system down even on an 8-core Mac Pro. I also wanted a multithreaded replacement for wget and curl, not some kludge of a script that runs multiple instances of these. So I searched further and found what I think right now is the ultimate most modern multithreaded CLI downloader there is -- aria2. The big problem I had with axel was that it 'faked' downloading files over SSL. I caught it doing that with tcdump. It was downloading https links as ordinary http. That really pissed me off and if I hadn't checked, I would have had a false sense of security. I doubt that many people know about this serious breach in security. Getting back to aria2, it is more advanced than any other downloader. It supports HTTP(S), FTP, BitTorrent, and Metalink protocols, is multiplatform, and is a download guerrilla. It maxes out my ISP's bandwidth with no load on the CPU or hard disk, unlike DTA. The man page is gigantic. I will never use more than a few of its many options. And oh, BTW, I checked its SSL performance with tcdump and it is solid, not fake. I wrote a script that mimics DTA's behavior, if not its convenience.

The basic command I use to get max bandwidth is

aria2c --file-allocation=none -c -x 10 -s 10 -d "mydir" URL

-c allows continuation of download if it gets interrupted, -x 10 and -s 10 allow up to 10 connections per server, and -d "mydir" outputs file to directory mydir.

aria2files.sh:

#!/bin/bash

filename="$1" # get filename from command line argument

while read -r line
do
    if [ "$line" ] # skip blank lines
    then
        if [[ "$line" =~ (https?|ftp)\:\/\/ ]] # line contains a URL, download file
        then
            echo "URL: '$line'"
            aria2c --file-allocation=none -c -x 10 -s 10 -d "$currdir" "$line"
        else # line contains a directory name, create directory if not already present
            echo "Directory: '$line'"
            currdir="$line"
            if [ ! -d "$currdir" ]
            then
                mkdir -p "$currdir" # '-p' enables creation of nested directories in one command
            fi
        fi
    fi
done < "$filename"

It reads a text file of the format:

files.txt:

dierctory 1
url1
url2
…
directory 2/subdirectory/sub-subdirectory/…
url3
url4
…
…
…

The script reads the filename from the command line:

aria2files.sh files.txt

It creates the directories and downloads to them. It can create nested directories as shown in the second example.

For more details see my post Bash script to download files from URLs to specified directories listed in a text file.

5
  • Can you apply your method here too askubuntu.com/q/813483/25388 My unsuccessful attempt aria2c -x10 -s10 http://horatio.cs.nyu.edu/mit/tiny/data/tiny_images.bin. Aug 17, 2016 at 18:29
  • 1
    Thanks @hmj6jmh! For the record on a Rapsberry Pi Model 3: wget -4 -c <https://URL/ISO_FILE> gives me ~40 KB/sec (87300K .......... .......... 11% 38.7K 4h28m when the same file downloaded with aria2c --disable-ipv6 -c gives ~250 KB/sec (144MiB/717MiB(20%) CN:1 DL:250KiB ETA:39m3s]).
    – tuk0z
    Mar 20, 2018 at 13:58
  • 6
    If you (like me) want to avoid Sourceforge, aria2 is part of the repositories, so you can install it with sudo apt install aria2
    – Bar
    Oct 23, 2018 at 13:44
  • Good answer. For sites that force HTTPS-only, axel straight-up doesn't work. It seems to have been updated to use HTTPS, but that version isn't in my repositories yet. aria2 worked well for me. Dec 1, 2018 at 21:34
  • To change default download behavior, you can provide those in aria2.conf file at $XDG_CONFIG_HOME/aria2/aria2.conf where XDG_CONFIG_HOME is typically at ~/.config/ for unix systems. See man aria2c for available options you can add to aria2.conf file.
    – Samir
    Jul 15 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.