Workflow with self made packages

Question

I have a folder ~/Packaging where all my public and private packages are in. For example, in the Packaging/think-rotate folder, there is the typical Debian stuff, like so:

think-rotate-2.5.2/
think-rotate-2.5.2.orig/
think-rotate-2.6/
think-rotate-2.6.orig/
think-rotate_1.0.tar.gz
think-rotate_2.5.2-0ubuntu1.debian.tar.gz
think-rotate_2.5.2-0ubuntu1.dsc
think-rotate_2.5.2-0ubuntu1_source.build
think-rotate_2.5.2-0ubuntu1_source.changes
think-rotate_2.5.2.orig.tar.gz@
think-rotate_2.5.2.tar.gz
think-rotate_2.5.orig.tar.gz@
think-rotate_2.5.tar.gz
think-rotate_2.6-0ubuntu1_all.deb
think-rotate_2.6-0ubuntu1_amd64.build
think-rotate_2.6-0ubuntu1_amd64.changes
think-rotate_2.6-0ubuntu1.debian.tar.gz
think-rotate_2.6-0ubuntu1.dsc
think-rotate_2.6-0ubuntu1_source.build
think-rotate_2.6-0ubuntu1_source.changes
think-rotate_2.6-0ubuntu1_source.stable.upload
think-rotate_2.6.orig.tar.gz@
think-rotate_2.6.tar.gz

I have some script that goes through my projects and generates the latest .tar.gz from my source code. So for the matter of this question, new tar archieves pop up in the packaging directory every now and then.

Since I do not want to run uupdate ../….tar.gz on every project by hand, I wrote a Python 3 script which takes care of that, runs debuild -S and dput to upload it to my Launchpad PPA, and then debuild to build it for my local machine.

The Debian changelog is just populated with the default “New upstream release” message. Those packages are more for organized local deployment, than for the general public.

The script then also uses apt-cache show to check whether the latest version is installed, and if not, uses dpkg -i to install that package.

This works somewhat, but I encountered a couple problems with my current version of that script:

• When I upgrade my machine to a new Ubuntu version, it does not rebuild everything. So I have a lot of packages that did not receive any upstream updates, and the latest version in my PPA is quantal, not raring.

• Packages are build for amd64 on my main computer, and often cannot be installed on my other computer, which has i686. Some packages are all, so that does not hurt. But I would need to rebuild the any packages on that machine. The Launchpad PPA takes care of building it for every architecture, but my script did not upload that package for raring, see first problem.

To summarize, the workflow has to contain all those steps:

• Extract new source tar archive and update the debian/changelog. (uupdate)

• If not already uploaded to PPA, build a source package and upload it to the Launchpad PPA. But only do this, if this is public package, do not upload my private packages to the PPA. (debuild -S and dput)

• If the package needs to be upgraded locally, build a binary package for the current architecture and install it. (debuild and debi or dpgk -i)

• Check if the series is outdated, i. e. the latest upload was for quantal, and the current system runs on raring. If so, rebuild the source again and upload to PPA.

Before you say that I should work on every package by hand, keep in mind that I have 43 packages sitting around, and I would really like to save work here.

I could work on my script to work around the current problems, but I'd rather use something that already works instead of rolling my own. Is there some tookchain or workflow that maintainers use in order to keep so many packages up to date in their PPAs or official repositories?

---

Update 2013-07-08: I now worked on the script to handle the problems as well. But I am still interested in a canonical solution.

Answer 1

I maintain quite a few packages at work, so let me give you my personal perspective on this.

All your stuff should be built in one central location, for all the distributions and architectures that you need. Do not upload source packages, and then build again locally. Pick your "private" packages from the build server where you host everything. That way, you can keep your private repositories in /etc/apt/sources.list with all the others (password protect them if needed).

If Launchpad does not work for you, set up your own build system. Since you have a lot of packages (over 40), setting up a build environment tailored to your special needs will pay off sooner or later.

You will need a dedicated machine with network access for that – either in your home or at work or a rented server in a datacenter.

Here are two options:

1. Use Jenkins CI and build script to do the package building. There is Jenkins Debian Glue by Michael Prokop, which does the job and it's well documented (I ended up rewriting it from scratch, but it might work for you). It's 100% shell code, so be prepared to do some Bash scripting if you need to tweak it.
   With this, you'll be able to build all your packages for all the combinations of dists and arches that you desire. Jenkins is wonderfully flexible and ROCK stable, and it will be easy enough to compartmentalize your repositories (for private and public consumption).

2. Use Buildbot – this is going to be more manual work than option one, but Buildbot is a Python project, and from your question I assume that you will feel at home setting that up. The i3 (window manager) project has an excellent description of how they set up their Buildbot environment, use that as a starting point.

I also strongly recommend sbuild over pbuilder. Jenkins Debian Glue uses pbuilder (one of the reasons why I ended up discarding it).

In any case, you will need to do some things that you did not have to, so far, such as setting up package repositories (which can be pretty painful) and maintaining a build server. At the end of the time, it will be worth it, though – and you will learn a lot.