29

We're a distributed team, which is why our VMs use the Ubuntu mirror:// setup. Our /etc/apt/sources.list looks like the following:

deb mirror://mirrors.ubuntu.com/mirrors.txt lucid main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt lucid-updates main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt lucid-backports main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt lucid-security main restricted universe multiverse

This by itself is pretty amazing and very helpful for people who work in different locations — less local customization, etc. necessary. Fail-over in theory.

In day-to-day, this setup fails too often. I want to say 2-3 times this week.

Right now mirrors.ubuntu.com returns ftp.uni-bayreuth.de as my closet mirror. Unfortunately, it seems to be down.

This has been going on for a few hours and the mirror is hosted by volunteers at a university and today being a Friday, my hopes are low that this is getting corrected soon.

All talk, my questions are:

  • Does anyone use this?
  • How do you work around downtime? (my quick-fix is a shell script)
  • How can I help to improve this situation?
6
  • 1
  • @RaduRădeanu - it is first time I read about netselect ... 8oþ Jul 18, 2013 at 15:31
  • @RaduRădeanu Interesting tip! Cheers! I am trying to use the mirror: method which had the most updates in your thread. My problem is that whenever the fastest mirror fails, then there is no failover and I am stuck.
    – Till
    Jul 18, 2013 at 15:46
  • Is creating Your Own Ubuntu Mirror an option?
    – Mitch
    Jul 20, 2013 at 10:01
  • @Till Can you let me know if you can have your own mirror.
    – Mitch
    Jul 21, 2013 at 14:13

6 Answers 6

10
+200

Personally I think that the best way to select the best Ubuntu repositories mirrors is to use the GUI method:

download server

Now, to improve the situation described in the question, you need to set some rules somehow. These rules must to act on mirrors.ubuntu.com. I can suggest some rules as follow:

  • make a list of best/preferred mirrors; there are a lot of mirrors as you can see here, here or here
  • if you found a good mirror, add it to the list
  • if a mirror was down or broken sometimes, it means that is not a good mirror and you should remove it from the list
  • you can use netselect, apt-spy or apt-fast
  • and others, depending on your requirements.

Next, to see how you can work around, I can give you a method described step by step with three bash scripts examples. First script use the mirrors from the country you are in at the moment instead of mirrors.ubuntu.com/mirrors.txt (for each country there is a text file with mirrors asociated; see http://mirrors.ubuntu.com/):

  • In a terminal run mkdir -p bin - this command will make a bin directory in your home folder if you don't already have it.
  • After run gedit ~/bin/change_sources.sh - this will create the new file change_sources.sh in gedit.
  • Copy and paste one of the next scripts in the new created file:
#!/bin/bash

export DISPLAY=:0

if ! [ "`ping -c 1 google.com`" ]; then
    notify-send "No internet connection"
    exit 0  
fi

ip=$(curl -s 'http://ipecho.net/plain')
country=$(curl -s 'http://geoiplookup.net/geoapi.php?output=countrycode' \
    | awk '{ print toupper($2) }')
release=$(lsb_release -sc)

file="/etc/apt/sources.list"
old_file="/etc/apt/sources.list.old"

line=$(head -n 1 $file)
new_line="## Ubuntu Repos for $ip"

if [ "$line" == "$new_line" ] ; then
    exit 0
fi

cp -f $file $old_file

printf "$new_line
deb mirror://mirrors.ubuntu.com/$country.txt $release main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/$country.txt $release-updates main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/$country.txt $release-backports main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/$country.txt $release-security main restricted universe multiverse
" > $file

notify-send "$file has been changed" "The old file has been put in $old_file"

exit 0

or, something similar to what can be found at http://repogen.simplylinux.ch/:

#!/bin/bash

export DISPLAY=:0

if ! [ "`ping -c 1 google.com`" ]; then
    notify-send "No internet connection"
    exit 0  
fi

ip=$(curl -s 'http://ipecho.net/plain')
country=$(curl -s 'http://geoiplookup.net/geoapi.php?output=countrycode' \
    | awk '{ print tolower($2) }')
release=$(lsb_release -sc)

file="/etc/apt/sources.list"
old_file="/etc/apt/sources.list.old"

line=$(head -n 1 $file)
new_line="## Ubuntu Main Repos for $ip"

if [ "$line" == "$new_line" ] ; then
    exit 0
fi

cp -f $file $old_file

printf "$new_line
deb http://$country.archive.ubuntu.com/ubuntu/ $release main restricted universe  multiverse
deb-src http://$country.archive.ubuntu.com/ubuntu/ $release main restricted universe multiverse

## Ubuntu Update Repos for $ip
deb http://$country.archive.ubuntu.com/ubuntu/ $release-security main restricted universe multiverse
deb http://$country.archive.ubuntu.com/ubuntu/ $release-updates main restricted universe multiverse
deb-src http://$country.archive.ubuntu.com/ubuntu/ $release-security main restricted universe multiverse
deb-src http://$country.archive.ubuntu.com/ubuntu/ $release-updates main restricted universe multiverse
" > $file

notify-send "$file has been changed" "The old file has been put in $old_file"

exit 0

or, a script using netselect (download from here, install instructions here) as izx explained verry nice in this answer:

#!/bin/bash

export DISPLAY=:0

if ! [ "`ping -c 1 google.com`" ]; then
    notify-send "No internet connection"
    exit 0  
fi

url=$(netselect \
    `wget -q -O- https://launchpad.net/ubuntu/+archivemirrors \
        | grep -P -B8 "statusUP|statusSIX" \
        | grep -o -P "(f|ht)tp.*\"" \
        | tr '"\n' '  '` \
    | awk '{print $2}')
release=$(lsb_release -sc)

if [ "$url" == "" ] ; then
    exit 0
fi

file="/etc/apt/sources.list"
old_file="/etc/apt/sources.list.old"

cp -f $file $old_file

printf "## Ubuntu Best Repos
deb http://extras.ubuntu.com/ubuntu $release main
deb-src http://extras.ubuntu.com/ubuntu $release main
deb $url $release main universe restricted multiverse
deb http://security.ubuntu.com/ubuntu/ $release-security restricted universe main multiverse
deb $url $release-updates restricted universe main multiverse
" > $file

notify-send "$file has been changed" "The old file has been put in $old_file"

exit 0
  • Save the file and close it.
  • Go back into terminal and run: chmod +x ~/bin/change_sources.sh - to grant execute access for the script.
  • Just for test, to run your new script, type in terminal ~/bin/change_sources.sh. It will give you an error, because you don't have the right to edit /etc/apt/sources.list. So, use sudo ~/bin/change_sources.sh
  • Edit the root user's crontab file using sudo crontab -e command and add the following line:
@hourly /home/$USER/bin/change_sources.sh  
#change $USER with your user name
  • I have set the cron job for every hour, but you can change as you wish or as you think is better. See http://en.wikipedia.org/wiki/Cron in this sense.
  • Save the file and check the new crontab entry with sudo crontab -l.

NOTE: To revert the changes made by this script, delete the cron job and follow the indications from the picture above or use next command in terminal:

cp -f /etc/apt/sources.list.bak /etc/apt/sources.list

From now, the file will be dynamically changed after it finds a change of the IP address.

It may not be the best solution, but, in my opinion, a good solution can be given in this way like in the scripts above.

9
  • 2
    I guess I'd argue if a gui on linux can be ever considered the traditional way. :) But anyway: Your first script solves a problem, I don't have. The mirror syntax already gets my German mirrors, or U.S. or wherever people are. The second script uses the country mirrors — they seem to have the least bandwidth available (saw that in the archivemirrors list on launchpad), the third script assumes that a fast (or the closest) mirror is the indeed the best. Back to square one. I'm coding a small apt-spy script right now. Will update here later.
    – Till
    Jul 20, 2013 at 11:46
  • 1
    @Till Well, your question were "How do you work around downtime?" and "How can I help to improve this situation?". I showed how I work around and I suggested you how can you help to improve... Also, if you read my answer until the end, you can see that I am aware that "It may not be the best solution, but I think that a good solution can be given in this way like in the scripts above". Finally, if you are interested to fix/improve broken servers, this seems off-topic here and you should speak with the owners of those servers instead. Jul 20, 2013 at 15:56
  • 2
    I don't think it's off-topic. Various Canonical/Ubuntu people told me to add my question here. It's why I am here in the first place and I guess also why there is a bounty assigned.
    – Till
    Jul 20, 2013 at 16:49
  • 1
    @Till Yep, the question is good, but not as you reformulated it in these comments. Nobody will answer at "how to fix a broken mirror server" here as far as is not an administrator at the respective server. Jul 20, 2013 at 17:01
  • 1
    I don't want to fix that server, I want to fix the selection algorithm and the lack of failover.
    – Till
    Jul 20, 2013 at 18:38
4

I appreciate all the input on this question, but since no one came up with a simple solution which fit our circumstances, I decided to fix the problem myself.

I created a tool (specifically for Ubuntu) which I call apt-spy2.

The primary objective of this tool is to find a working mirror fast. Working is defined by that the mirror server is available and (hopefully :) up to date.

I make no assumptions about if the selected server is necessarily the closest and the fastest. I'm not doing any pings or GEO DNS tricks — but so far this works when something breaks.

How it works — in a nutshell:

  1. I use http://mirrors.ubuntu.com or launchpad's list of mirrors to retrieve servers.
  2. I do a simple check on each (for HTTP Response Status Code).
  3. LBNL, I update /etc/apt/sources.list.

Please note: This assumes that people play nice and put additional mirrors (e.g. 3rd party repositories into /etc/apt/sources.list.d. But I guess that means there's room for improvement.

You can obtain this tool like so:

$ [sudo] gem install apt-spy2

The cli comes with list, check, fix and help (with extended information on how to use it).

I tried to document as much as possible in the project's README.

The current version is a very conservative 0.5.0.

The code is open source and the license is liberal. And I take all contributions.

5
  • Did you uploaded this to ubuntu repositories?
    – Braiam
    Mar 29, 2014 at 19:03
  • @Braiam You can obtain this tool like so: sudo gem install apt-spy2 Mar 29, 2014 at 19:43
  • @RaduRădeanu I'm asking if he uploaded it, since I'm interested in packaging something to Debian.
    – Braiam
    Mar 29, 2014 at 20:01
  • @Braiam I didn't package it up, but I would welcome it if someone did it. :)
    – Till
    Apr 2, 2014 at 22:31
  • @Braiam Btw, this is unlikely to work with Debian. ;( Only works with Ubuntu.
    – Till
    Apr 4, 2014 at 16:17
2

There was this command at Debian 6.0.4 :

apt-spy

This did the job to find the next closest available server automatic and to generate a new sources.list

In Ubuntu this command seems not to exist ?

It still exists in Debian 7.0 wheezy :

https://launchpad.net/debian/wheezy/+source/apt-spy/+copyright

You can download your *.deb package here :

http://packages.debian.org/sid/apt-spy

... still searching for the sources ...

You obviously need Debian-7.0-Installation for to get the source-code after editing sources-list with entry :

deb-src http://http.debian.net/debian wheezy main

Then after sudo apt-get update you would simply suck the code with :

sudo apt-get source apt-spy

7
  • Thanks for the suggestion, apt-spy looks great indeed. I'm not sure that works well. Every time I've installed debian stuff into ubuntu, it broke sooner or later due to other dependencies.
    – Till
    Jul 18, 2013 at 14:54
  • @Till - it is always something with bit more digging ... as soon as Linux is getting more demanding. Jul 18, 2013 at 15:40
  • 1
    Build it for ubuntu: rubygems.org/gems/apt-spy2 and github.com/lagged/apt-spy2
    – Till
    Jul 20, 2013 at 21:32
  • @Till Do you mean: Build it ...! or Built it ...? - I look after next time - now is full-moon here and am groggy ... Jul 20, 2013 at 22:56
  • you are quick ! Jul 20, 2013 at 22:58
2

Maybe not using the solution you like, but setting up a local cache for the .deb files should be ideal when you download several MB of package (mostly of them I bet are the same) and since you are downloading from the same server, you might like to sacrifice a PC to make the cache and make the server less busy.

Procedure

You should be able to set apt-cacher in a machine that have lots of space in the Hard Drive (you would need it). You can configure this using the Apt-cacher server guide, but lets do a quick resume.

  1. Install apt-cacher

    sudo apt-get install apt-cacher apache2

  2. Edit /etc/default/apt-cacher and set the autostart value to 1, so your cache starts when the machine power up.
  3. Restart apache sudo /etc/init.d/apache2 restart
  4. Test your cache using http://ip-of.your.cache:3142/apt-cacher.
  5. You should allow host to use the cache, so edit /etc/apt-cacher/apt-cacher.conf and look for the line allowed_hosts. You can set subnets like 192.168.0.0/24 or just * for all hosts
  6. Look for a fast and reliable server around. You could use apt-spy for this (this command could take hours since it also test the speed of each server).
  7. In your clients you have two options:

    • set the sources.list as follow:

    deb http://ip-of.your.cache:3142/the.server.you.like/ubuntu/ lucid main restricted universe multiverse

    Change the ip-of.your.cache for the ip of your cache and the.server.you.like for the host name of the server you will be using. You can use several lines.

    • Or set apt to use a proxy: edit /etc/apt/apt.conf.d/01proxy and add the line:

    Acquire::http::Proxy "http://ip-of.your.cache:3142";

4
  • If you would use apt-spy worldwide it would last a while ... but you can use this command with bordering to e.g. Europe by : apt-spy -d testing -a Europe Jul 21, 2013 at 18:33
  • @dschinn1001 -d testing smells like Debian.
    – Braiam
    Jul 21, 2013 at 18:50
  • well then : apt-spy -a Europe Jul 22, 2013 at 0:37
  • "downvote" ? - I did no downvote ? - it happens sometimes that I click by accident with my mouse - I never do down-vote with aim ! so voted you up ! thx. Jul 22, 2013 at 10:52
1

You can try setting up your own Ubuntu mirror. That way updates will be local.

apt-mirror needs a high speed internet connection and a lot of disk space

Setting up your own Ubuntu mirror is very easy by using apt-mirror.

1) go into the apt-mirror repository. Back up your sources.list file like so:

sudo cp /etc/apt/sources.list /etc/apt/sources.list.old

2) Edit the sources.list file like so:

gksudo gedit /etc/apt/sources.list

3) And insert the following on a new line:

deb http://apt-mirror.sourceforge.net/ apt-mirror

4) Let’s update the package index

sudo apt-get update

5) And install apt-mirror

sudo apt-get install apt-mirror

6) Apt-mirror is now set up so let’s tell it what servers to mirror:

sudo cp /etc/apt/mirror.list /etc/apt/mirror.list.old
gksudo gedit /etc/apt/mirror.list

The default configuration will put your downloaded files in the /var/spool/apt-mirror folder. Here’s how the mirror.list file looks like:

############# config ##################
#
set base_path /media/silo/repo
#
# if you change the base path you must create the directories below with write privlages
#
# set mirror_path $base_path/mirror
# set skel_path $base_path/skel
# set var_path $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch <running host architecture>
set nthreads 20
set tilde 0
#
############# end config ##############
deb http://archive.ubuntu.com/ubuntu lucid main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid -updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid -backports main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid -security main restricted universe multiverse
clean http://archive.ubuntu.com/ubuntu

this only mirrors binary packages but if you want source packages, you should insert the appropriate lines. They usually take the form:

deb-src http://gh.archive.ubuntu.com/ubuntu/ lucid main restricted

7) Now, run apt-mirror. This is going to take quite a while. Don’t worry if you cannot download all the files in one go, apt-mirror can resume downloads (kill it with the Ctrl+C key combination and re-run it when you want to continue). Okay, run apt-mirror like so:

sudo apt-mirror /etc/apt/mirror.list

Also check out How To Create A Local Debian/Ubuntu Mirror With apt-mirror

Source:Use Apt-mirror to Create Your Own Ubuntu Mirror

6
  • If memory serves me right, apt-mirror is available in ubuntu repositories.
    – Braiam
    Jul 22, 2013 at 15:32
  • BTW, make sure to note the need of lots of space in the HD to download a complete mirror ;)
    – Braiam
    Jul 22, 2013 at 15:57
  • Thanks for the answer and it's a great resource for setting up a mirror and I've considered that, but then I'd need this setuo in at least 3 different locations. I know I am picky here, but I still feel that mirrors.ubuntu.com ought to work. :)
    – Till
    Jul 22, 2013 at 16:49
  • @till mirrors.ubuntu.com will never resolve mirrors.ubuntu.com, but a server that is nearest to you. Problem is that server it resolve is an unreliable one. Your better play is using a reliable server from the official list.
    – Braiam
    Jul 22, 2013 at 23:48
  • @Braiam any idea why unreliable mirrors are returned by it then?
    – Till
    Jul 23, 2013 at 17:53
1

I've written a Python script called apt-select to serve as a CLI alternative to the GUI method.

The script queries each mirror via TCP, prints to stdout mirrors with the lowest latency, and generates a new sources.list file. There are also multiple options that allow you to choose from a list of ranked mirrors, and retrieve information such as when the mirror was last updated and its bandwidth capacity.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .