axel upon Gufran's recommendation but it hugely disappointed me. My goal was to find a CLI replacement for
DownThemAll because it hogs the CPU and hard disc and slows the entire system down even on an 8-core Mac Pro. I also wanted a multithreaded replacement for
curl, not some kludge of a script that runs multiple instances of these. So I searched further and found what I think right now is the ultimate most modern multithreaded CLI downloader there is -- aria2. The big problem I had with
axel was that it 'faked' downloading files over SSL. I caught it doing that with
tcdump. It was downloading
https links as ordinary
http. That really pissed me off and if I hadn't checked, I would have had a false sense of security. I doubt that many people know about this serious breach in security. Getting back to
aria2, it is more advanced than any other downloader. It supports HTTP(S), FTP, BitTorrent, and Metalink protocols, is multiplatform, and is a download guerrilla. It maxes out my ISP's bandwidth with no load on the CPU or hard disk, unlike DTA. The man page is gigantic. I will never use more than a few of its many options. And oh, BTW, I checked its SSL performance with
tcdump and it is solid, not fake. I wrote a script that mimics DTA's behavior, if not its convenience.
The basic command I use to get max bandwidth is
aria2c --file-allocation=none -c -x 10 -s 10 -d "mydir" URL
-c allows continuation of download if it gets interrupted,
-x 10 and
-s 10 allow up to 10 connections per server, and
-d "mydir" outputs file to directory
filename="$1" # get filename from command line argument
while read -r line
if [ "$line" ] # skip blank lines
if [[ "$line" =~ (https?|ftp)\:\/\/ ]] # line contains a URL, download file
echo "URL: '$line'"
aria2c --file-allocation=none -c -x 10 -s 10 -d "$currdir" "$line"
else # line contains a directory name, create directory if not already present
echo "Directory: '$line'"
if [ ! -d "$currdir" ]
mkdir -p "$currdir" # '-p' enables creation of nested directories in one command
done < "$filename"
It reads a text file of the format:
The script reads the filename from the command line:
It creates the directories and downloads to them. It can create nested directories as shown in the second example.
For more details see my post Bash script to download files from URLs to specified directories listed in a text file.