TL;DR version: I'm setting up a secure apt repository with signed Release file, and it's giving me a "Failed to fetch Release" error message, which goes away entirely when I remove the Release.gpg file. I can confirm that apt is getting the Release file even when it says it's unable to.
My organization has a need for a local secure-apt repository, where we will host our own copy of updates for Ubuntu 12.04. The need comes from a number of reasons (bandwidth, pre-testing of updates before releasing to production, all the usual.) In the process of building the repo, I've run into an unusual problem.
First, the basics. The apt repo is being hosted on a small desktop running CentOS 6.5, not a server; currently it's proof of concept. I'm using nginx to act as a web server, and an Ubuntu 12.04 VM to act as a client test bed for the proof of concept. I made a backup copy of /etc/apt/sources.list, then deleted it and started from scratch. It currently contains only:
cat /etc/apt/sources.list # Test repository for local private apt deb http://mbwinnc641090/ precise main
The directory structure has been copied from the existing Ubuntu apt repos for Precise. The Release and Release.gpg files sit in /dists/precise (where / is relative to the root of the web server, not the file server); Packages, Packages.gz and Packages.bz2 sit in /dists/precise/main/binary-amd64. There is a Packages in /dists/precise/main/binary-i386; it does not appear in Release.gpg, it is zero bytes in size, and was only added to solve a secondary problem that cropped up while I was looking at the real issue (will explain in more detail below.) I generated a 4096 RSA/4096 RSA key with OpenGPG, and have signed the release file with:
gpg -abs -o Release.gpg Release
I have exported the public key with
gpg --armor --output ./APT-key-2.key --export <myid>
copied it onto a USB stick and then brought that over to the VM; the key imported successfully onto the Ubuntu VM with
apt-key add ./APT-key-2.key
The Release file contents are as follows:
Origin: Ubuntu Label: Ubuntu Suite: precise Version: 12.04 Codename: precise Date: Wed, 25 Apr 2012 22:49:23 UTC Architectures: amd64 Components: main restricted universe multiverse Description: Ubuntu Precise 12.04 MD5Sum: 3a72e2cbeccb018e855e59851c97fafb 1806 main/binary-amd64/Packages 4e8e2385e95080b1c6e61ea032f286af 1023 main/binary-amd64/Packages.bz2 871b13afb486f3faf4174e3c839ad6da 938 main/binary-amd64/Packages.gz SHA1: d58e402014879a2d7f9a8890144daf8d05216bc0 1806 main/binary-amd64/Packages 5782287bf8d853da47425dbf15aaf924a4505e1d 1023 main/binary-amd64/Packages.bz2 c469ea854a1f15638fcff418e4ca527bb3a12a52 938 main/binary-amd64/Packages.gz SHA256: ec3f7712fecf94de2337a0c554219f370ed2ddb4486e1e1cc147f90fb4b5d765 1806 main/binary-amd64/Packages 5776bcdc534063094459f283619c85861758bae3718273ce6e34b6e052f9a495 1023 main/binary-amd64/Packages.bz2 5c0eedc77ff872f2676c79116e55182e759cb8c41285a49bae075a7ea08f9942 938 main/binary-amd64/Packages.gz
As you can see, it contains MD5Sum, SHA1 and SHA256 records for the main/binary-amd64/Packages, Packages.bz2 and Packages.gz files. I have double- and triple-checked that these are correct.
The Package files themselves contain only one package record (again, proof of concept - start small and work up) for Firefox. It's lengthy and I do not believe it's relevant to the problem, so I will omit it for now, but if someone wants to see it I will edit it in afterwards.
Now, the actual problem - when apt-get is run, it complains about failing to fetch the Release file:
sudo apt-get -s --allow-unauthenticated update [sudo] passwd for administrator: Hit http://mbwinnc641090 precise Release.gpg Hit http://mbwinnc641090 precise Release W: Failed to fetch http://mbwinnc641090/dists/precise/Release Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)
However, I can confirm with 100% certainty that it is not failing to pull the Release file down. First, the nginx log files confirm that the apt request is getting both files; access.log reports either HTTP 304 and 0 byte transfers (i.e. no change since the last copy of the file acquired), or if I touch both files on the repo, HTTP 200 and the transfer sizes of both files is correct:
10.113.226.222 - - [21/Feb/2014:08:12:12 -0600] "GET /dists/precise/Release.gpg HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.14)" "-" 10.113.226.222 - - [21/Feb/2014:08:12:12 -0600] "GET /dists/precise/Release HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.14)" "-"
10.113.226.222 - - [21/Feb/2014:10:05:04 -0600] "GET /dists/precise/Release.gpg HTTP/1.1" 200 836 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.14)" "-" 10.113.226.222 - - [21/Feb/2014:10:05:04 -0600] "GET /dists/precise/Release HTTP/1.1" 200 966 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.14)" "-"
Both files are being transferred as plaintext; I added the "default_type text/plain" directive to the conf file for the nginx server (note - whole server, not just this one virtual host.)
I can also confirm that apt-get is receiving the files - /var/lib/apt/lists has both mbwinnc641090_dists_precise_Release and mbwinnc641090_dists_precise_Release.gpg; the file contents match exactly what is on the server; the file size is exactly what is on the server (and what is transferred according to nginx); and the file timestamp changes to match the server time when I touch both files on the server. wget also succeeds in accessing the files, and I can browse directly to them in Chrome/Firefox/etc. So they're there and available.
Where it gets really weird is that when I remove Release.gpg from the directory (either move it to another directory or rename it), suddenly the apt-get update command works. There's nothing for it to update (and initially it did complain about not being able to find Packages in /dists/precise/main/binary-i386, which went away when I created the empty Packages file there), but the error message about not being able to remove Release goes away entirely and it exits with zero error code. So it looks like something about apt does not like the signature I have. However, I can confirm that gpgv is able to verify the signature using the apt keyring:
sudo gpgv --keyring /etc/apt/trusted.gpg mbwinnc641090_dists_precise_Release.gpg mbwinnc641090_dists_precise+Release gpgv: Signature made Fri 21 Feb 2014 10:04:56 AM CST using RSA key ID AF57B06B gpgv: Good Signature from "<my info>"
Running apt get with "-o Debug::Acquire::gpgv=true" further confirms that Apt is passing the key validation stage - apt-get returns:
Hit http://mbwinnc641090 precise Release.gpg Hit http://mbwinnc641090 precise Release 33% [Release gpgv 966 B]inside VerifyGetSigners gpgv path: /usr/bin/gpgv Keyring file: /etc/apt/trusted.gpg Keyring path: /etc/apt/trusted.gpg.d/ Preparing to exec: /usr/bin/gpgv /usr/bin/gpgv --ignore-time-conflict --status-fd 3 --keyring /etc/apt/trusted.gpg /var/lib/apt/lists/partial/mbwinnc641090_dists_precise_Release.gpg /var/lib/apt/lists/mbwinnc641090_dists_precise_Release Read: [GNUPG:] SIG_ID sLGwzAAcePo9LZtAdgNf44grYaQ 2014-02-21 1392998696 Read: [GNUPG:] GOODSIG 84332B63AF57B06B <myinfo> Got GOODSIG, key ID:GOODSIG 84332B63AF57B06B Read: [GNUPG:] VALIDSIG 09DBAF28ACFD21DC5FBF797784332B63AF57B06B 2014-02-21 1392998696 0 4 0 1 2 00 09DBAF28ACFD21DC5FBF797784332B63AF57B06B gpgv exited gpgv succeeded W: Failed to fetch http://mbwinnc641090/dists/precise/Release Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file) E: Some index files failed to download. They have been ignored, or old ones used instead.
So basically - I'm out of ideas. Any help at all would be appreciated; I'm sure at this point it's just some incredibly stupid simple thing that I've missed and you'll all have a good laugh, but I'm completely hosed.