1

I am trying to keep an offline machine (16.04 amd64 laptop) up to date by mirroring the Ubuntu software repositories. Unfortunately, I have a number of constraints beyond my control:

  1. Host machine (connected to internet) is CentOS 7. I don't have apt-mirror, debmirror, or admin rights. I do have wget, rsync, and I have compiled lftp
  2. Though I do have rsync, the rsyncd.service is not running.

    $ systemctl list-unit-files | grep rsync
    ...
    rsyncd.service                                disabled
    ...
    

    So I'm not sure I can even use rsync using the instructions at the Rsyncmirror documentation according to this answer.

  3. Portable drive is only 1TB with ~850Gb free. I don't need any source code from the repos, just binaries, but even still, do I even have enough space?
  4. Slow network. Not really a constraint, but an annoyance. I have the ability to let this download go over a weekend or as long as it takes, but would prefer something that supports parallel downloads.

What I tried:

  1. rsync -a --bwlimit=128 rsync://archive.ubuntu.com/ubuntu /media/mirror/ubuntu This fails with the same errors as shown in the shown in this answer. I cannot start rsynd.service without admin, so I think this is a no go.
  2. Naive wget --recursive http://se.archive.ubuntu.com/ubuntu/. This pulls in way too much stuff and is pretty slow. Not sure of any way to filter out files I don't need or do any parallel downloading.
  3. lftp. I think this is showing the most promise at the moment. For instance:

    $ lftp http://mirror.math.princeton.edu/pub/ubuntu/
    cd ok, cwd=/pub/ubuntu
    lftp mirror.math.princeton.edu:/pub/ubuntu> mirror -n -parallel=10
    

    works great, but of course is also just going to pull in everything. I can exclude whatever I want with --exclude-globs (e.g. --exclude-glob "*trusty*/" --exclude-glob "*artful*/" --exclude-glob "*bionic*/" --exclude-glob "*precise*/" --exclude-glob "*.tar.gz"), but what should I exclude to trim the size while still getting the contents to build a working repository.

I know that after I am done getting all of the packages I am going to have to actually make this a working repo by using dpkg-scanpackages or something similar. I can do that from the offline laptop using Ubuntu/Debian tools, so that doesn't concern me as much. I really just want to get some advice on how to only download the packages which are relevant for my machine.

1 Answer 1

0

I realized it might make more sense to go with an additive approach rather than a subtractive one; that is, get a list of all the files/packages I need and then download only those ones (instead of trying to filter all of the ones I don't want).

The package listings are stored in Packages.gz, so I wrote a script to download all of the compressed package listings I want, extract the package names from the listings, compose this list into an lftp script, and then launch lftp.

I think that would be enough at a bare minimum, but I also sync up a few other folders with some metadata which seems important.

#!/bin/sh
remote="http://mirror.math.princeton.edu"
dists=( xenial xenial-security xenial-updates)
repos=( main restricted universe multiverse)
package_lists=()

printf "open %s \ncd /pub/ubuntu/\n" "$remote" > sync-mirror.lftp

echo "mirror -n -v -O ubuntu/indices -f indices/md5sums.gz" >> sync-mirror.lftp

for dist in "${dists[@]}"; do
    for repo in "${repos[@]}"; do
        pkg_lists+=("$remote/pub/ubuntu/dists/$dist/$repo")
        echo "mirror -n -v --parallel=10 -O ubuntu/indices -f indices/override.$dist.$repo" >> sync-mirror.lftp
        echo "mirror -n -v --parallel=10 -O ubuntu/dists/$dist/$repo -F dists/$dist/$repo/binary-amd64" >> sync-mirror.lftp
        echo "mirror -n -v --parallel=10 -O ubuntu/dists/$dist/$repo -F dists/$dist/$repo/dep11" >> sync-mirror.lftp
        echo "mirror -n -v --parallel=10 -O ubuntu/dists/$dist/$repo -F dists/$dist/$repo/i18n" >> sync-mirror.lftp
    done
done

for pkg_list in "${pkg_lists[@]}"; do
    curl "$pkg_list/binary-amd64/Packages.gz" | \
        gunzip | \
        grep Filename | \
        sed 's|Filename: \(.*\)|mirror -n -v --parallel=10 -O ubuntu/\1 -f \1 |g' >> sync-mirror.lftp

done

printf "\n exit\n" >> sync-mirror.lftp

 lftp -f sync-mirror.lftp

I would like to just feed mirror a list of files and tell it to download in parallel, but I have not found a way to do that. I think the current method is taking longer than it should because of all of the individual calls to mirror. Also, I'm not sure the --parallel flag is doing anything in conjunction with -f since it is only getting a single file.

Running the script over the weekend and will test when I get back. Will update answer with results.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .