0

TL;DR version: I'm setting up a secure apt repository with signed Release file, and it's giving me a "Failed to fetch Release" error message, which goes away entirely when I remove the Release.gpg file. I can confirm that apt is getting the Release file even when it says it's unable to.

My organization has a need for a local secure-apt repository, where we will host our own copy of updates for Ubuntu 12.04. The need comes from a number of reasons (bandwidth, pre-testing of updates before releasing to production, all the usual.) In the process of building the repo, I've run into an unusual problem.

First, the basics. The apt repo is being hosted on a small desktop running CentOS 6.5, not a server; currently it's proof of concept. I'm using nginx to act as a web server, and an Ubuntu 12.04 VM to act as a client test bed for the proof of concept. I made a backup copy of /etc/apt/sources.list, then deleted it and started from scratch. It currently contains only:

cat /etc/apt/sources.list

# Test repository for local private apt

deb http://mbwinnc641090/ precise main

The directory structure has been copied from the existing Ubuntu apt repos for Precise. The Release and Release.gpg files sit in /dists/precise (where / is relative to the root of the web server, not the file server); Packages, Packages.gz and Packages.bz2 sit in /dists/precise/main/binary-amd64. There is a Packages in /dists/precise/main/binary-i386; it does not appear in Release.gpg, it is zero bytes in size, and was only added to solve a secondary problem that cropped up while I was looking at the real issue (will explain in more detail below.) I generated a 4096 RSA/4096 RSA key with OpenGPG, and have signed the release file with:

gpg -abs -o Release.gpg Release

I have exported the public key with

gpg --armor --output ./APT-key-2.key --export <myid>

copied it onto a USB stick and then brought that over to the VM; the key imported successfully onto the Ubuntu VM with

apt-key add ./APT-key-2.key

The Release file contents are as follows:

Origin: Ubuntu
Label: Ubuntu
Suite: precise
Version: 12.04
Codename: precise
Date: Wed, 25 Apr 2012 22:49:23 UTC
Architectures: amd64
Components: main restricted universe multiverse
Description: Ubuntu Precise 12.04
MD5Sum:
3a72e2cbeccb018e855e59851c97fafb        1806 main/binary-amd64/Packages
4e8e2385e95080b1c6e61ea032f286af        1023 main/binary-amd64/Packages.bz2
871b13afb486f3faf4174e3c839ad6da         938 main/binary-amd64/Packages.gz
SHA1:
d58e402014879a2d7f9a8890144daf8d05216bc0        1806 main/binary-amd64/Packages
5782287bf8d853da47425dbf15aaf924a4505e1d        1023 main/binary-amd64/Packages.bz2
c469ea854a1f15638fcff418e4ca527bb3a12a52         938 main/binary-amd64/Packages.gz
SHA256:
ec3f7712fecf94de2337a0c554219f370ed2ddb4486e1e1cc147f90fb4b5d765        1806 main/binary-amd64/Packages
5776bcdc534063094459f283619c85861758bae3718273ce6e34b6e052f9a495        1023 main/binary-amd64/Packages.bz2
5c0eedc77ff872f2676c79116e55182e759cb8c41285a49bae075a7ea08f9942         938 main/binary-amd64/Packages.gz

As you can see, it contains MD5Sum, SHA1 and SHA256 records for the main/binary-amd64/Packages, Packages.bz2 and Packages.gz files. I have double- and triple-checked that these are correct.

The Package files themselves contain only one package record (again, proof of concept - start small and work up) for Firefox. It's lengthy and I do not believe it's relevant to the problem, so I will omit it for now, but if someone wants to see it I will edit it in afterwards.

Now, the actual problem - when apt-get is run, it complains about failing to fetch the Release file:

sudo apt-get -s --allow-unauthenticated update
[sudo] passwd for administrator:

Hit http://mbwinnc641090 precise Release.gpg
Hit http://mbwinnc641090 precise Release
W: Failed to fetch http://mbwinnc641090/dists/precise/Release  Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)

However, I can confirm with 100% certainty that it is not failing to pull the Release file down. First, the nginx log files confirm that the apt request is getting both files; access.log reports either HTTP 304 and 0 byte transfers (i.e. no change since the last copy of the file acquired), or if I touch both files on the repo, HTTP 200 and the transfer sizes of both files is correct:

No touch:

10.113.226.222 - - [21/Feb/2014:08:12:12 -0600] "GET /dists/precise/Release.gpg HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.14)" "-"
10.113.226.222 - - [21/Feb/2014:08:12:12 -0600] "GET /dists/precise/Release HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.14)" "-"

Touch:

10.113.226.222 - - [21/Feb/2014:10:05:04 -0600] "GET /dists/precise/Release.gpg HTTP/1.1" 200 836 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.14)" "-"
10.113.226.222 - - [21/Feb/2014:10:05:04 -0600] "GET /dists/precise/Release HTTP/1.1" 200 966 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.14)" "-"

Both files are being transferred as plaintext; I added the "default_type text/plain" directive to the conf file for the nginx server (note - whole server, not just this one virtual host.)

I can also confirm that apt-get is receiving the files - /var/lib/apt/lists has both mbwinnc641090_dists_precise_Release and mbwinnc641090_dists_precise_Release.gpg; the file contents match exactly what is on the server; the file size is exactly what is on the server (and what is transferred according to nginx); and the file timestamp changes to match the server time when I touch both files on the server. wget also succeeds in accessing the files, and I can browse directly to them in Chrome/Firefox/etc. So they're there and available.

Where it gets really weird is that when I remove Release.gpg from the directory (either move it to another directory or rename it), suddenly the apt-get update command works. There's nothing for it to update (and initially it did complain about not being able to find Packages in /dists/precise/main/binary-i386, which went away when I created the empty Packages file there), but the error message about not being able to remove Release goes away entirely and it exits with zero error code. So it looks like something about apt does not like the signature I have. However, I can confirm that gpgv is able to verify the signature using the apt keyring:

sudo gpgv --keyring /etc/apt/trusted.gpg mbwinnc641090_dists_precise_Release.gpg mbwinnc641090_dists_precise+Release
gpgv: Signature made Fri 21 Feb 2014 10:04:56 AM CST using RSA key ID AF57B06B
gpgv: Good Signature from "<my info>"

Running apt get with "-o Debug::Acquire::gpgv=true" further confirms that Apt is passing the key validation stage - apt-get returns:

Hit http://mbwinnc641090 precise Release.gpg
Hit http://mbwinnc641090 precise Release
33% [Release gpgv 966 B]inside VerifyGetSigners
gpgv path: /usr/bin/gpgv
Keyring file: /etc/apt/trusted.gpg
Keyring path: /etc/apt/trusted.gpg.d/
Preparing to exec: /usr/bin/gpgv /usr/bin/gpgv --ignore-time-conflict --status-fd 3 --keyring /etc/apt/trusted.gpg /var/lib/apt/lists/partial/mbwinnc641090_dists_precise_Release.gpg /var/lib/apt/lists/mbwinnc641090_dists_precise_Release
Read: [GNUPG:] SIG_ID sLGwzAAcePo9LZtAdgNf44grYaQ 2014-02-21 1392998696
Read: [GNUPG:] GOODSIG 84332B63AF57B06B <myinfo>
Got GOODSIG, key ID:GOODSIG 84332B63AF57B06B
Read: [GNUPG:] VALIDSIG 09DBAF28ACFD21DC5FBF797784332B63AF57B06B 2014-02-21 1392998696 0 4 0 1 2 00 09DBAF28ACFD21DC5FBF797784332B63AF57B06B
gpgv exited
gpgv succeeded
W: Failed to fetch http://mbwinnc641090/dists/precise/Release  Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)

E: Some index files failed to download.  They have been ignored, or old ones used instead.

So basically - I'm out of ideas. Any help at all would be appreciated; I'm sure at this point it's just some incredibly stupid simple thing that I've missed and you'll all have a good laugh, but I'm completely hosed.

1 Answer 1

1

The problem is very, very subtle.

All of the hash lines need a space in front of them. For example, this:

MD5Sum:
3a72e2cbeccb018e855e59851c97fafb        1806 main/binary-amd64/Packages
4e8e2385e95080b1c6e61ea032f286af        1023 main/binary-amd64/Packages.bz2
871b13afb486f3faf4174e3c839ad6da         938 main/binary-amd64/Packages.gz

needs to be:

MD5Sum:
 3a72e2cbeccb018e855e59851c97fafb        1806 main/binary-amd64/Packages
 4e8e2385e95080b1c6e61ea032f286af        1023 main/binary-amd64/Packages.bz2
 871b13afb486f3faf4174e3c839ad6da         938 main/binary-amd64/Packages.gz
1
  • Oh. My. God. That's the most frustrating problem I've run into in my whole life. Thank you for finding that for me; I'd more or less given up. I even saw the spaces in front of the hash lines in the public repos, but the idea that leading whitespace would be necessary never even dawned on me. Jun 16, 2014 at 19:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .