0

The context is a bug in which Ubuntu are updating package list timestamps every 30 minutes on their archive mirrors, which forces [apt-get|aptitude] update commands to download package lists though they haven't changed:

The proposed workaround is to run touch over the affected index files prior to running apt-get update. This make their timestamps current, and avoids having to re-download them. This speeds processing by up to several minutes (2-3 seconds vs. 2-6 minutes in various tests). This despite running a local apt-cache-ng proxy.

We're running into this testing (mostly) and deploying (somewhat) Ubuntu-based images for our AWS infrastructure in which apt-get update runs a couple of times. 3-6 minutes vs. 3-20 seconds makes a big difference in our iteration time.

It seems odd to me that apt doesn't rely on filesize or hashes to determine whether or not files have changed. Timestamps are very trivially modified, and seem awfully brittle to base this behavior on.

Improve this question
3
  • You shouldn't need to run apt-get update every thirty seconds, so I don
    – ionFish
    Jun 28, 2012 at 21:43
  • 1
    The suggestion to 'touch' things isn't a proposed 'fix'. It is a hacky work-around. A much better work-around is suggested, use a HTTP proxy.
    – Zoredache
    Jun 28, 2012 at 22:05
  • ionFish: I've just edited my question to address that. We're testing AWS deployments and the issue comes up there. Zoredache: Noted, edited in question. And, actually, we're using apt-cacher-ng. I've confirmed watching both the client and the cache that we're hitting the origin.
    – Dr. Edward Morbius
    Jun 28, 2012 at 22:21

1 Answer 1

Reset to default
1

When I first tested the touch hack, I was surprised that apt-get used the local file mod time to inform the http server what to download--but that's obviously the case (or the hack wouldn't work).

The downloaded Release file contains a reliable date and these files are crytographically signed, and are therefore quite trustworthy, but the hashes aren't used to determine what package files to download.

With the timestamp approach now in use for package files, there's nothing special needed from the mirror site. We owe the Debian project for the design. It's helpful to mirror owners to keep Ubuntu identical to Debian and keep things vanilla and therefore easy to manage.

This design normally works fine, as far as I know. This is just a bug which should be fixed as far as I'm concerned, but the bug is in Canonical private code so outsiders like me can't help patch it. I hope they do so soon as we are abusing the hospitality of those running Ubuntu mirrors, in addition to causing unnecessary expense for some Ubuntu users who must pay for bandwidth by the gigabyte.

There is an http protocol mechanism for using an etag to represent a served item, but it's not used for apt-get. Such a tag could contain the hash from the Release file.

It's the large number of packages included in Ubuntu releases which make this bug so significant.

From the signed (and therefore checksummed) Release, there are hashes for these files, and in these files hashes for versions of packages and from there checksums for files. Since the package files contain the version numbers, and since they don't change, the bug doesn't cause redundant package upgrades.

Improve this answer
3
  • 1
    How do you suggest that the ETags would get updated in a way that applies to all web servers? Keep in mind, that not everybody is running Apache, you need a extremely basic web server to support APT repositories.
    – Zoredache
    Jun 29, 2012 at 0:01
  • @Zordache -- I'm not proposing it, just trying to say the protocol mechanism exists in case the questioner had that in the back of his mind.
    – John S Gruber
    Jun 29, 2012 at 0:06
  • @Zordache -- may I suggest that you post an answer to the apt-get list question above containing your proxy and rsync approaches? I understand that Debian was or is considering a file delta approach to file size issues, but that Ubuntu wasn't as of last month.
    – John S Gruber
    Jun 29, 2012 at 0:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .