1

I am trying to keep an offline machine (16.04 amd64 laptop) up to date by mirroring the Ubuntu software repositories. Unfortunately, I have a number of constraints beyond my control:

  1. Host machine (connected to internet) is CentOS 7. I don't have apt-mirror, debmirror, or admin rights. I do have wget, rsync, and I have compiled lftp
  2. Though I do have rsync, the rsyncd.service is not running.

    $ systemctl list-unit-files | grep rsync
    ...
    rsyncd.service                                disabled
    ...
    

    So I'm not sure I can even use rsync using the instructions at the Rsyncmirror documentation according to this answer.

  3. Portable drive is only 1TB with ~850Gb free. I don't need any source code from the repos, just binaries, but even still, do I even have enough space?
  4. Slow network. Not really a constraint, but an annoyance. I have the ability to let this download go over a weekend or as long as it takes, but would prefer something that supports parallel downloads.

What I tried:

  1. rsync -a --bwlimit=128 rsync://archive.ubuntu.com/ubuntu /media/mirror/ubuntu This fails with the same errors as shown in the shown in this answer. I cannot start rsynd.service without admin, so I think this is a no go.
  2. Naive wget --recursive http://se.archive.ubuntu.com/ubuntu/. This pulls in way too much stuff and is pretty slow. Not sure of any way to filter out files I don't need or do any parallel downloading.
  3. lftp. I think this is showing the most promise at the moment. For instance:

    $ lftp http://mirror.math.princeton.edu/pub/ubuntu/
    cd ok, cwd=/pub/ubuntu
    lftp mirror.math.princeton.edu:/pub/ubuntu> mirror -n -parallel=10
    

    works great, but of course is also just going to pull in everything. I can exclude whatever I want with --exclude-globs (e.g. --exclude-glob "*trusty*/" --exclude-glob "*artful*/" --exclude-glob "*bionic*/" --exclude-glob "*precise*/" --exclude-glob "*.tar.gz"), but what should I exclude to trim the size while still getting the contents to build a working repository.

I know that after I am done getting all of the packages I am going to have to actually make this a working repo by using dpkg-scanpackages or something similar. I can do that from the offline laptop using Ubuntu/Debian tools, so that doesn't concern me as much. I really just want to get some advice on how to only download the packages which are relevant for my machine.

0

I realized it might make more sense to go with an additive approach rather than a subtractive one; that is, get a list of all the files/packages I need and then download only those ones (instead of trying to filter all of the ones I don't want).

The package listings are stored in Packages.gz, so I wrote a script to download all of the compressed package listings I want, extract the package names from the listings, compose this list into an lftp script, and then launch lftp.

I think that would be enough at a bare minimum, but I also sync up a few other folders with some metadata which seems important.

#!/bin/sh
remote="http://mirror.math.princeton.edu"
dists=( xenial xenial-security xenial-updates)
repos=( main restricted universe multiverse)
package_lists=()

printf "open %s \ncd /pub/ubuntu/\n" "$remote" > sync-mirror.lftp

echo "mirror -n -v -O ubuntu/indices -f indices/md5sums.gz" >> sync-mirror.lftp

for dist in "${dists[@]}"; do
    for repo in "${repos[@]}"; do
        pkg_lists+=("$remote/pub/ubuntu/dists/$dist/$repo")
        echo "mirror -n -v --parallel=10 -O ubuntu/indices -f indices/override.$dist.$repo" >> sync-mirror.lftp
        echo "mirror -n -v --parallel=10 -O ubuntu/dists/$dist/$repo -F dists/$dist/$repo/binary-amd64" >> sync-mirror.lftp
        echo "mirror -n -v --parallel=10 -O ubuntu/dists/$dist/$repo -F dists/$dist/$repo/dep11" >> sync-mirror.lftp
        echo "mirror -n -v --parallel=10 -O ubuntu/dists/$dist/$repo -F dists/$dist/$repo/i18n" >> sync-mirror.lftp
    done
done

for pkg_list in "${pkg_lists[@]}"; do
    curl "$pkg_list/binary-amd64/Packages.gz" | \
        gunzip | \
        grep Filename | \
        sed 's|Filename: \(.*\)|mirror -n -v --parallel=10 -O ubuntu/\1 -f \1 |g' >> sync-mirror.lftp

done

printf "\n exit\n" >> sync-mirror.lftp

 lftp -f sync-mirror.lftp

I would like to just feed mirror a list of files and tell it to download in parallel, but I have not found a way to do that. I think the current method is taking longer than it should because of all of the individual calls to mirror. Also, I'm not sure the --parallel flag is doing anything in conjunction with -f since it is only getting a single file.

Running the script over the weekend and will test when I get back. Will update answer with results.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.