How do I optimize apt-fast with Amazon Web Services (AWS) mirrors?

Question

I know that Amazon Web Services has hosted mirrors that I can use to update my Ubuntu machines.

I know there is a tool called apt-fast that makes multiple connections to a mirror server to improve speed, but I'm concerned that increasing this kind of load on a mirror server isn't polite, but I still want a fast mirror, how can I optimize apt-fast with AWS?

Answer 1

apt-fast works in the same way common "download accelerators" work. It opens more connections than normal to the web server and then combines them all together for a speed boost.

Most sysadmins I know prevent this sort of thing by limiting the amount of concurrent connections from the same IP, afterall, if you're increasing the amount of connections to your computer, that's less bandwidth for other people.

However while investigating using Amazon's Ubuntu mirrors I found out that AWS is actually optimized for high connections, with lower overall throughput. So in other words, AWS optimizes serving HTTP totally opposite from what you'd think.

1. First install apt-fast:

   sudo add-apt-repository ppa:apt-fast/stable
   sudo apt-get update
   sudo apt-get install apt-fast
   

   Feel free to go through the config prompts, we're going to adjust them anyway.

2. Configure your system to use Amazon's mirrors.

3. Configure apt-fast to use more connections, by editing /etc/apt/apt-fast.conf:

   1. Find the MIRRORS line, and add every region you want to add as an additional mirror. Basically it'll hit these repositories in addition to what's in your sources list, mine looks like this:

      MIRRORS=( 'us-east-1.ec2.archive.ubuntu.com/ubuntu,us-east-2.ec2.archive.ubuntu.com/ubuntu,us-west-1.ec2.archive.ubuntu.com/ubuntu' )
      

   2. Find the _MAXNUM= line, and adjust to the number of concurrent connections, I'm using:

      _MAXNUM=35
      

4. Save the file, then you can use the apt-fast command in place of apt-get, or you can just alias it.

Things I've discovered:

• I'm in the US midwest, for me a mirror mix of us-east and us-west works best, adding any other region does nothing to improve my speed. So you probably want to start with Amazon regions closest to you.
• 35 connections seems to be the sweet spot, anything lower is slower, anything higher seems to be higher latency and doesn't improve overall throughput; like anything else, test what works for you.
• Don't set this up for a normal ubuntu mirror, a really high connection count can be bad, even though most competent mirror admins will deny the extra connections anyway.
• I've only tested this remotely, since AWS mirrors in each region are basically LAN speed to the individual instances, then you shouldn't need to do this for your AWS instances.
• If you're interested in improving your overall performance and you have more than 2 servers/desktops locally, considering caching apt downloads instead or running your own local mirror.