Repro:
Install CentOS 7 server, then deploy Ubuntu 18.04 environment via debootstrap:
yum install epel-release
yum install debootsrap
mkdir /opt/ubuntu-18.04
ln -s /opt/ubuntu-18.04 /opt/ubuntu
debootstrap --arch=amd64 bionic /opt/ubuntu http://de.archive.ubuntu.com/ubuntu/
Enter chroot:
chroot /opt/ubuntu
Install apt-mirror:
apt install software-properties-common
add-apt-repository universe
apt install apt-mirror
Set up /etc/apt/mirror.list for multiple architectures. bionic-updates and bionic-backports, and also restricted, universe and multiverse are left out to speed up testing:
set nthreads 20
set _tilde 0
deb-i386 http://archive.ubuntu.com/ubuntu bionic main
# restricted universe multiverse
deb-amd64 http://archive.ubuntu.com/ubuntu bionic main
# restricted universe multiverse
deb-arm64 http://ports.ubuntu.com/ubuntu-ports bionic main
# restricted universe multiverse
#deb-i386 http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse
#deb-amd64 http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse
#deb-arm64 http://ports.ubuntu.com/ubuntu-ports bionic-updates main restricted universe multiverse
#deb-i386 http://archive.ubuntu.com/ubuntu bionic-backports main restricted universe multiverse
#deb-amd64 http://archive.ubuntu.com/ubuntu bionic-backports main restricted universe multiverse
#deb-arm64 http://ports.ubuntu.com/ubuntu-ports bionic-backports main restricted universe multiverse
clean http://archive.ubuntu.com/ubuntu
clean http://ports.ubuntu.com/ubuntu-ports
Perform the sync:
sudo -u apt-mirror apt-mirror
Set up your HTTP server of choice (I use nginx), the directory structure looks like this:
# ls -al /var/www/html
total 62100
drwxr-xr-x. 3 nginx nginx 4096 Jul 2 13:54 .
drwxr-xr-x. 3 nginx nginx 4096 Jul 1 20:10 ..
drwxr-xr-x. 2 root root 4096 Jul 1 17:42 seed
lrwxrwxrwx. 1 root root 65 Jul 1 20:10 ubuntu -> /opt/ubuntu/var/spool/apt-mirror/mirror/archive.ubuntu.com/ubuntu
-rw-r--r--. 1 root root 63564717 Jul 1 17:43 ubuntu-18.04-netinstall.zip
lrwxrwxrwx. 1 root root 69 Jul 2 13:54 ubuntu-ports -> /opt/ubuntu/var/spool/apt-mirror/mirror/ports.ubuntu.com/ubuntu-ports
Set up /etc/apt/sources.list on your Ubuntu 18.04 desktop client. Same things are left out, but they behave the same way, with -updates and -backports it'll cause 3 files missing and returning 404:
# Security
deb [arch=amd64,i386] http://security.ubuntu.com/ubuntu bionic-security main restricted universe multiverse
deb [arch=arm64] http://ports.ubuntu.com/ bionic-security main restricted universe multiverse
# Internal mirror
deb [arch=amd64,i386] https://apt.foobar.com/ubuntu bionic main
# restricted universe multiverse
#deb [arch=amd64,i386] https://apt.foobar.com/ubuntu bionic-updates main restricted universe multiverse
#deb [arch=amd64,i386] https://apt.foobar.com/ubuntu bionic-backports main restricted universe multiverse
deb [arch=arm64] https://apt.foobar.com/ubuntu-ports bionic main
# restricted universe multiverse
#deb [arch=arm64] https://apt.foobar.com/ubuntu-ports bionic-updates main restricted universe multiverse
#deb [arch=arm64] https://apt.foobar.com/ubuntu-ports bionic-backports main restricted universe multiverse
# Canonical fallback mirror
deb [arch=amd64,i386] http://archive.ubuntu.com/ubuntu bionic main restricted universe multiverse
deb [arch=amd64,i386] http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse
deb [arch=amd64,i386] http://archive.ubuntu.com/ubuntu bionic-backports main restricted universe multiverse
deb [arch=arm64] http://ports.ubuntu.com/ bionic main restricted universe multiverse
deb [arch=arm64] http://ports.ubuntu.com/ bionic-updates main restricted universe multiverse
deb [arch=arm64] http://ports.ubuntu.com/ bionic-backports main restricted universe multiverse
Now perform apt update (same result with apt-get update):
# apt update
Get:1 https://apt.foobar.com/ubuntu bionic InRelease [242 kB]
Get:2 https://apt.foobar.com/ubuntu-ports bionic InRelease [242 kB]
Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:4 http://ports.ubuntu.com bionic-security InRelease
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88,7 kB]
Hit:6 http://ports.ubuntu.com bionic InRelease
Hit:7 http://ports.ubuntu.com bionic-updates InRelease
Hit:8 http://ports.ubuntu.com bionic-backports InRelease
Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74,6 kB]
Get:10 https://apt.foobar.com/ubuntu-ports bionic/main arm64 Packages [975 kB]
Get:11 http://security.ubuntu.com/ubuntu bionic-security InRelease [88,7 kB]
Get:12 https://apt.foobar.com/ubuntu-ports bionic/main Translation-en [516 kB]
Get:13 https://apt.foobar.com/ubuntu-ports bionic/main amd64 DEP-11 Metadata [477 kB]
Get:13 https://apt.foobar.com/ubuntu-ports bionic/main amd64 DEP-11 Metadata [477 kB]
Ign:13 https://apt.foobar.com/ubuntu-ports bionic/main amd64 DEP-11 Metadata
Ign:13 https://apt.foobar.com/ubuntu-ports bionic/main amd64 DEP-11 Metadata
Ign:13 https://apt.foobar.com/ubuntu-ports bionic/main amd64 DEP-11 Metadata
Err:13 https://apt.foobar.com/ubuntu-ports bionic/main amd64 DEP-11 Metadata
404 Not Found [IP: 10.1.3.96 443]
Fetched 494 kB in 1s (659 kB/s)
Reading package lists... Done
E: Failed to fetch https://apt.foobar.com/ubuntu-ports/dists/bionic/main/dep11/Components-amd64.yml 404 Not Found [IP: 10.1.3.96 443]
E: Some index files failed to download. They have been ignored, or old ones used instead.
There was a hint on #ubuntu that it's because the ubuntu-ports/dists/bionic/Release file lists Components-amd64.yml. Now, the problem with that is that it's listed in the official ports.ubuntu.com file too, and that folder structure doesn't have Components-amd64.yml either, yet, apt update only gives the 404 errors for the local mirror.
We've been using this local mirror just fine with amd64/i386 packages on a few clients already, it's only the arm64 repo files that give this error.
What am I doing wrong? Why is apt looking for amd64 files on an arm64 repo?
Update: after messing around with debmirror I got the exact same issue. At this point it became suspicious that it's something on the client side. After digging into /etc/apt I've found the following excerpt in /etc/apt/apt.conf.d/50appstream:
# The YAML metadata files
deb::DEP-11 {
MetaKey "$(COMPONENT)/dep11/Components-$(NATIVE_ARCHITECTURE).yml";
ShortDescription "Components-$(NATIVE_ARCHITECTURE)";
Description "$(RELEASE)/$(COMPONENT) $(NATIVE_ARCHITECTURE) DEP-11 Metadata";
KeepCompressed "true";
KeepCompressedAs "gz";
};
Try this:
sed -i 's/NATIVE_ARCHITECTURE/ARCHITECTURE/' /etc/apt/apt.conf.d/50appstream
And the apt update problems are gone. Can anyone confirm it's indeed an Ubuntu bug and this is indeed the solution to it?
