163

I have multiple Ubuntu machines at home and a pretty slow internet connection, and sometimes multiple machines need to be updated at once (especially during new Ubuntu releases.)

Is there a way where only one of my machines needs to download the packages, and the other machines can use the first machine to get the debs? Does it involve setting up my own local mirror? Or a proxy server? Or can it be made simpler?

0

6 Answers 6

135

I did some research into a bunch of solutions and some Ubuntu developers came up with a proxy configuration (based on Squid) for 10.04 and later. It's called squid-deb-proxy. It only requires a machine to act as the server. Large organizations usually run their own full mirrors but for most people the on demand mirroring is enough.

Why squid-deb-proxy?

  • No editing of files on the client side.
  • Use zeroconf so that clients were "zero config"
  • Use an existing, solid proxy solution instead of writing a new tool.
  • Easy to set up for a typical Linux administrator.

Server Config

On the machine you want to act as a server install the tool with:

sudo apt-get install squid-deb-proxy avahi-utils

Now start the service bits:

 sudo start squid-deb-proxy

And the avahi bits (You don't need this if you're on 12.04+):

 sudo start squid-deb-proxy-avahi

This will install the proxy server (which listens to port 8000 by default) and the avahi tools needed for the server to advertise itself on your network via zeroconf.

Client Config

On each of the computers that you want to use the cache (the clients, and the server itself so it can use the cache too), you need to install the client side tool with:

sudo apt install squid-deb-proxy-client

Optional: For maximum efficiency you should set one machine to automatically download updates, so that when your other machines need it it's already in the cache. You can do this by going to System->Administration->Update Manager, then click on the "Settings..." button, in the Update tab set it to automatically download all the updates.

alt text

Caching 3rd Party Sources

By default the cache is set up to only cache official Ubuntu repositories. To add more you need to add them to the list of sources at /etc/squid-deb-proxy/mirror-dstdomain.acl. This is where you can add ppa.launchpad.net, or other services you might use. After making changes to this file, you must run sudo restart squid-deb-proxy in order for the changes to be effective.

Manual Config

If for some reason you do not want to use zeroconf (for network reasons or whatever), you can manually set a client to use the proxy by editing /etc/apt/apt.conf and adding the following stanza, (replace the 0.0.0.0 with the IP address of the server):

 Acquire { 
   Retries "0"; 
   HTTP { Proxy "http://0.0.0.0:8000"; };
 };

Firewall

In case you are using a firewall, avahi uses 5353 over addresses 224.0.0.0/4 and requires a rule that looks like this:

# Specifically port 5353 which avahi uses
-A INPUT -i eth2 -d 224.0.0.0/4 --dport 5353 -j ACCEPT

# OR

# Wide open so all local broadcasting works
-A INPUT -i eth2 -d 224.0.0.0/4 -j ACCEPT

Next, you need to open TCP port 8000 for the actual communication through the proxy. Something more or less like this:

-A INPUT -i eth2 -p tcp -m tcp --dport 8000 -d 192.168.0.1 -s 192.168.0.0/24 --syn -j ACCEPT

These rules are just to help you. They will probably not match your setup one to one. (i.e. wrong interface, wrong private network IP addresses, etc.)

Confirming it Works

First tail the log on the server so you can look at it: tail -F /var/log/squid-deb-proxy/access.log and then run an update on any machine that has the client installed; the log should start to scroll with entries like this:

1307310795.647     32 192.168.1.106 TCP_MISS/302 768 GET http://us.archive.ubuntu.com/ubuntu/dists/natty-proposed/universe/i18n/Translation-en.xz - DIRECT/141.210.26.10 text/html
1307310795.683     34 192.168.1.106 TCP_MISS/302 752 GET http://us.archive.ubuntu.com/ubuntu/dists/natty/main/i18n/Translation-en_US.lzma - DIRECT/141.210.26.10 text/html
1307310795.716     32 192.168.1.106 TCP_MISS/302 746 GET http://us.archive.ubuntu.com/ubuntu/dists/natty/main/i18n/Translation-en.lzma - DIRECT/141.210.26.10 text/html
1307310795.750     32 192.168.1.106 TCP_MISS/302 764 GET http://us.archive.ubuntu.com/ubuntu/dists/natty/multiverse/i18n/Translation-en_US.lzma - DIRECT/141.210.26.10 text/html
1307310795.784     32 192.168.1.106 TCP_MISS/302 758 GET http://us.archive.ubuntu.com/ubuntu/dists/natty/multiverse/i18n/Translation-en.lzma - DIRECT/141.210.26.10 text/html
1307310795.817     32 192.168.1.106 TCP_MISS/404 657 GET http://us.archive.ubuntu.com/dists/natty-proposed/multiverse/i18n/Translation-en_US.xz - DIRECT/141.210.26.10 text/html

Which means the clients see the cache but are missing it, which is expected since it hasn't cached anything yet. Each subsequent run should show up as TCP_HIT. You can find the squid cache files themselves in /var/cache/squid-deb-proxy.

Using it

From then on all the machines on your network will check the cache before hitting the outside network to fetch packages. If there are new packages available then the first machine will download it from the net, after that subsequent requests for that package will come from the server to the clients.

TODO

We still need to enable apt to just use an advertised cache on the network out of the box and by default so you don't need to install the client piece. We also need to fix the bug that 403's deb's not in the mirror list.

6
  • 2014/01/21 14:56:31| ERROR: /var/cache/squid-deb-proxy/03: (2) No such file or directory FATAL: Failed to verify one of the swap directories, Check cache.log for details. Run 'squid -z' to create swap directories if needed, or if running Squid for the first time. Squid Cache (Version 3.3.8): Terminated abnormally.
    – thumper
    Jan 21, 2014 at 2:05
  • 1
    You need to also install squid.
    – thumper
    Jan 21, 2014 at 2:11
  • It looks like the Manual Config is not valid anymore. There is no apt.conf file under /etc/apt. However, there is a plethora of files with settings under /etc/apt/apt.conf.d. I would imagine that we should now create a file there instead? Jul 24, 2016 at 18:06
  • Also, it looks like the TODO has been resolved. That is, I did not have to do anything and the cache was used first. Although the bug does not look like it was marked as fixed yet. Jul 24, 2016 at 18:14
  • Sure, if apt.conf.d is how it's done now please submit an edit to this this answer, cheers! Jul 24, 2016 at 18:36
40

apt-cacher-ng is the answer for me - I haven't encountered any problems in smallish environments (approx. 20 clients), so I suppose the issues @MagicFab mentions were solved in current version (installed on Ubuntu 10.04 and 10.10). There is no config necessary for the server, and you only need to instruct your clients to use the server as their package manager proxy.

The server is completely installed and configured by installing the apt-cacher-ng package.

The clients need to be configured by setting up APT proxy - by adding the file /etc/apt/apt.conf.d/01proxy, containing this (where "your-apt-server" is your server's name or IP address):

Acquire::http { Proxy "http://your-apt-server:3142"; };

Done - now the packages will be cached by the server, no matter what sources you use or what system version you have (e.g. a 10.04 server can be used by 9.10,10.04 and 11.04 clients without any problems or conflicts).


If you have client laptop(s) that roam between networks, it gets a bit more complex - I've made a script that sets the right proxy depending on the network address; the script is executable and in /etc/network/if-up.d/apt-proxy. Upon receiving an IPv4 address from a DHCP server, the script will set the right apt-cacher server for the respective network:

#!/bin/sh

set -e
# Don't bother when lo is configured.
if [ "$IFACE" = lo ]; then
    exit 0
fi
# Only run from ifup.
if [ "$MODE" != start ]; then
    exit 0
fi
# currently only cares about IPv4
if [ "$ADDRFAM" != inet ] && [ "$ADDRFAM" != NetworkManager ]; then
    exit 0
fi
# only run for DHCP-assigned addresses
if [ "$DHCP4_IP_ADDRESS" = "" ]; then
    exit 0
fi

# we're matching on network *broadcast* address,
#  not the specific IP address we were assigned
case "$DHCP4_BROADCAST_ADDRESS" in
    10.3.141.255)
        PROXY='Acquire::http::Proxy "http://my-home-server:3142";';
        ;;
    192.168.154.255)
        PROXY='Acquire::http::Proxy "http://work-server.foo.bar.example.com:3142";';
        ;;
    # add as needed
    *)
        # unknown, no proxying
        PROXY=""
        ;;
esac

# set the proxy
FNAME="/etc/apt/apt.conf.d/01proxy"
echo -n "$PROXY">$FNAME

exit 0
5
  • apt-cacher-ng also supports IPv6 (I remember some of the other solutions didn't).
    – Azendale
    Jun 22, 2011 at 13:34
  • 12
    A year later, I've gone over to squid-deb-proxy and squid-deb-proxy-client: zeroconf indeed means zero configuration on clients, which is extremely useful for nomadic users - and in case a server doesn't respond, the client defaults to direct download. May 21, 2012 at 9:22
  • apt-cacher-ng sucks, it's verry buggy, hangs on downloading packages. for example i saw it downloaded 150 MB just for a 30MB package
    – pylover
    Sep 2, 2015 at 22:46
  • apt-cacher-ng works perfectly. You have something very wrong with YOUR system. On Precise squid-deb-proxy wants to install Squid. Insists on it.
    – Ken Sharp
    Nov 20, 2015 at 19:00
  • Now that is the best solution to get a cache working with pbuilder. This worked out of the box. Also it looks like there is support for zeroconf (in 2016), but I could not (yet) get that to work on my end. Jul 24, 2016 at 21:46
7

I much prefer setting up a local mirror using the debmirror utility.

Here is an example incantation.

debmirror --progress --verbose --nosource --method=ftp --passive \
 --host=ftp.osuosl.org --root=pub/ubuntu \
 --dist=lucid,lucid-updates,lucid-security,lucid-backports \
 --section=main,restricted,universe,multiverse --arch=amd64 \
 /d2/ftp/mirror/ubuntu-lucid

I run this about once a week, and use it as the basis for establishing one or more "patchlevels". For example...

 cd /d2/ftp/mirror/
 cp -al ubuntu-lucid ubuntu-lucid-20100908

This creates a linked copy of the tree (uses almost zero disk space) which I can point each of my local servers to in apt sources.list

1
  • 5
    You should check out ubumirror, it's a convenience package in the archive that has all that already set up for you. It's what the official mirrors use. Sep 8, 2010 at 17:39
6

One of the easiest solution is to setup apt-proxy.

Read the ubuntu documentation here : https://help.ubuntu.com/community/AptProxy

2

In small networks (such as home / small office), I've used apt-cacher-ng with good results. I haven't checked the latest versions, but I know it needs careful setup of both server and clients, and it's best suited for clients that will only get updates from your local network.

I tried the squid-based solution above but it required applying several workaround and more client configuration than I'd like to, so it doesn't feel yet like it could replace apt-cacher-ng in small setups.

1

apt-cacher wasn't the easiest to set up and it won't survive a dist-upgrade.

Install squid-deb-proxy on the server, squid-deb-proxy-client on the clients. It uses zeroconf Avahi, so no configuration necessary.

If you're looking to cache more than just debs, I wouldn't bother with Squid. Apache Traffic Server is the next big thing. http://trafficserver.readthedocs.org

1
  • 1
    squid-deb-proxy insists on installing squid. And apt-cacher is long dead, apt-cacher-ng is where it's at.
    – Ken Sharp
    Nov 20, 2015 at 19:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .