apt-cacher-ng fails to import from new distribution, ie bionic 18.04

Question

i'm running apt-cacher-ng (acng) on an ubuntu 16.04 server (desktop) with a slow network connection. i have an 18.04 laptop and have used a fast connection to download all installed packages, and rsync'd them to _import for acng. i then point the laptop apt to use the server port 3142 as a proxy (and verified that it is actually used). this should populate the acng index files. however, when i trigger the import from the server (http://localhost:3142/acng-report.html), none of the bionic files are imported. further, it appears that the index files (Package.xz) are never stored in the acng dir (they appear for older distributions, but not for bionic)

what needs to be done to import 18.04 packages on a 16.04 acng ? how are the index files being recreated on the laptop without showing up in the acng cache ?

# on laptop
echo 'Acquire::http { Proxy "http://desktop:3142"; };' | sudo tee /etc/apt/apt.conf.d/01proxy
sudo rm /var/cache/apt/*.bin
sudo rm /var/lib/apt/lists/*
sudo apt-get update

no hits are found during the import and at the bottom it says:

4908 files (2.1 GiB) left behind

Answer 1

i ended up figuring out how to do this. apt-cacher-ng can in theory import packages placed in _import, but modern apt doesn't appear to download the index files in a way that acng (at least the 0.9.1 version in ubuntu 16.04) recognizes so acng can't identify the imports and does nothing

the workaround is to explicitly request the index files, and then trigger the import using the webapp

# for bionic, with myhost as the acng host:

base="http://us.archive.ubuntu.com/ubuntu/dists"
for kk in bionic{,-updates,-backports}; do
  for jj in amd64 i386; do
    for ii in main multiverse restricted universe; do
      url="$base/$kk/$ii/binary-$jj/Packages.xz";
      echo $url;
      curl -x myhost:3142 $url > /dev/null;
    done;
  done;
done

# then trigger the import

at this point, everything imported successfully. i tracked this down using a variety of tools, which might be useful if you're trying to import from a different repository:

sudo strace -s 222 -v -y -yy -o ~/apt.log apt-get update
sudo ngrep -d any -vv -w byline outbound and not dst myhost or dst port 3142 > ~/apt2.log 2>&1

# strace output at the point of printing the package file

read(7<pipe:[341008]>, "200 URI Start...\nURI: http://us.archive.ubuntu.com/ubuntu/dists/bionic/universe/binary-amd64/by-hash/SHA256/39ec12bac1f788ae649d32c138898d49fdec15eee020a9c72ae92a622fb662a0\nSize:"..., 64000) = 232
write(1</dev/pts/4>, "Get:8 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [8,570 kB]\n", 84) = 84
stat("/var/lib/apt/lists/partial/us.archive.ubuntu.com_ubuntu_dists_bionic_universe_binary-amd64_Packages.xz", ...) = 0

# the ngrep output showed the same hash

so instead of downloading the Package.xz file directly, apt is using the hash (and apparently acng doesn't now about it). grepping for this hash:

sudo grep -r 39ec12ba... /var/cache/apt* /var/lib/apt*

/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_bionic_InRelease
39ec12ba...          8569560 universe/binary-amd64/Packages.xz

which is how apt knows to use the hash instead of the Packages.xz url