Debian archive rebuilds on Amazon Web Services

I like to think that archive rebuilds play an important role in Debian Quality Assurance and Release Management efforts. By trying to rebuild every Debian package from source, one can identify packages that do not build anymore due to changes in other packages (compilers, interpreters, libraries, …). It is also a good way to stress-test all packages that are involved in building other packages.

Since 2007, I had been running Debian archive rebuilds on the Grid’5000 testbed, a research infrastructure for performing experiments on distributed systems – HPC/Grid/Cloud/P2P. I filed more than 6000 release-critical bugs in the process.

Late last year, Amazon kindly offered us a grant to allow us to run such QA tests on Amazon Web Services. With S├ębastien Badia, we ported the rebuild infrastructure to AWS (scripts), and several rebuilds have already been carried out on AWS.

On the technical level, 50 to 100 EC2 spot instances are started, and then controlled from a master instance using SSH. On build instances, a classic sbuild setup is used. Logs are retrieved to the master node after rebuilds, and build instances are simply shut down when there are no more tasks to process. Several tasks are processed simultaneously on each instance, and when they fail, they are retried again with no other concurrent build on the same instance, to eliminate random failures caused by load or timing issues. All the scripts are designed to support other kind of QA tests, not just rebuilds.

Moving to Amazon Web Services will facilitate sharing the human workload of doing those tests. It is now possible for developers interested in custom tests to do them themselves (hint hint).

Going to RMLL (LSM) and Debconf!

Next week, I’ll head to Strasbourg for Rencontres Mondiales du Logiciel Libre 2011. On monday morning, I’ll be giving my Debian Packaging Tutorial for the second time. Let’s hope it goes well and I can recruit some future DDs!

Then, at the end of July, I’ll attend Debconf again. Unfortunately, I won’t be able to participate in Debcamp this year, but I look forward to a full week of talks and exciting discussions. There, I’ll be chairing two sessions about Ruby in Debian and Quality Assurance.

#debian-ubuntu on OFTC

If you are a Debian developer and need realtime interaction with an Ubuntu developer about the state of your packages in Ubuntu (or vice-versa), #debian-ubuntu on might be useful. I had forgotten about that channel, but it resurfaced during the discussions about improving communication between both projects.

EtherPad: web-based collaborative editor

I recently (during a UDS lightning talk) discovered EtherPad. It’s a collaborative editor (like gobby), but uses a browser instead of a standalone application. It’s free software (Google open sourced it after buying the company that was developing it), and there’s a free online instance at Setting up a new pad is as simple as going to and clicking Create Pad. It’s written in Java, and not packaged in Debian (yet).

Apply for the Google Summer of Code at Debian even if you are already a Debian Developer!

The Debian GSOC admins have made it clear that it is possible for DDs to be selected as GSOC students (even if people who are not Debian contributors will be prioritized). So, if you are a student, a Debian Developer, and interested in getting paid to work on Debian during the summer, it’s a very good idea to apply! For the details, see this blog post.

Note: the goal of this post is not to start a flamewar, but to make sure that everybody is fully aware of the unrestricted selection process. Since we are going to accept DDs as students anyway, let’s at least try to get the best applications.

Debian’s KVM + Ubuntu karmic => bug?

I’ve been playing with virtualization, and KVM in particular. However, I’m running into an interesting problem. Below is how Ubuntu Netbook Remix look inside my KVM (either using virt-viewer, virt-manager or directly KVM to display the VM). Note how the top panel is fine. I’m using KVM 88 from experimental (but I had the same problem with KVM 85 from unstable), the cirrus video driver inside the VM, and an up-to-date karmic VM.
Has someone ran into that problem already? If yes, where is it tracked? I’m been failing to find the correct search keywords so far.


PTS as a great tool for upstream developers

I recently realized how nice the Packages Tracking System is for upstream developers who want to follow the status of their software in Debian. By subscribing, they can easily monitor (and reply to) bug reports, and the general status of their software in Debian. I’m trying to help with coreutils maintainance, and it’s great to see upstream developers answer directly to bug reporters on Debian.

So, maintainers, tell your upstreams about the PTS! ;)

It might be a good idea to actually mention that use case on obvious places (not sure what the obvious places are, though).

mailing list for cross-distributions collaboration

A few months ago, I asked a set of questions on development mailing lists of a few GNU/Linux distributions. This resulted in very interesting discussions. As promised back then, all the answers from all distros I contacted can be read on the web or as an mbox file.

Also, kindly agreed to host a mailing list to ease discussions between distributions, and act as a central point of contact. You can subscribe, and post to distributions at lists dot freedesktop dot org.

This mailing list is for people involved (or interested) in the development of distributions. Questions that are on-topic are both technical and social/organizational issues, like:

  • How do you achieve graphical boot in your distro? Do you use some kind of dependancy-based or events-based boot?
  • How do you package both ruby 1.8, ruby 1.9 and jruby, or handle KDE vs KDE4?
  • Do you use a system that gives a limited set of rights to new contributors?

Off-topic stuff obviously include trolling about which distribution is the best one, or user support.

Don’t hesitate to forward this announcement to all interested parties. Let’s make this mailing list something useful together!

Also, I really apologize for procastinating announcing this list for sooo long. I’m really good at procastinating interesting stuff, it seems.

dd + /dev/random fun

From Adrien Lebre, who seems to be having fun working on kDFS:

$ dd if=/dev/random of=file bs=1024 count=2
0+2 records in
0+2 records out
256 bytes (256 B) copied, 0.000705717 s, 363 kB/s
$ ls -l file
-rw-r--r-- 1 lucas lucas 256 2008-02-08 17:53 file

Can be easily explained using strace:

open("/dev/random", O_RDONLY|O_LARGEFILE) = 0
open("file", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 1
read(0, "p(\361\241,\360-\330~M\245y\10=\274U\201c\207\v\336a\273"..., 1024) = 128
write(1, "p(\361\241,\360-\330~M\245y\10=\274U\201c\207\v\336a\273"..., 128) = 128
read(0, "\362,LW5l.?\242\22\252\223\206\375\10\326\335\316\374\372"..., 1024) = 128
write(1, "\362,LW5l.?\242\22\252\223\206\375\10\326\335\316\374\372"..., 128) = 128
close(0)                                = 0
close(1)                                = 0

Re: a problem with tools

Joey has a problem with tools used over dpkg-dev’s core tools to help maintainers maintain their debian packages.

I was wondering about the adoption of those tools, so I simply grepped the logs from my last rebuild for “^Setting up $TOOL “, which is more accurate than looking at build-deps, since someteam-pkg-tools could depend on cdbs, and hide cdbs usage.

Results, with lists of packages “affected”:

Tool Packages
debhelper 11434 (95.8%)
cdbs 2665 (22.3%)
dpatch 1714 (14.4%)
quilt 935 (7.8%)
dbs 57 (0.47%)
yada 32 (0.27%)
debmake 2 (0.017%)

(if you want me to check the logs for another tool, just tell me)

My personal opinion on this is that tools that make sense are slowly getting adopted (that probably includes debhelper, cdbs, dpatch and quilt). The main difference between debhelper and the other tools is that the other tools don’t try to solve a problem for everybody. cdbs perfectly makes sense for some packages: the simple ones, where you only have to modify 4-5 lines in the output from dh_make to build properly.

Patch systems are also a good way to track changes made to upstream source, and probably encourage giving back patches to upstream. Note that my stats don’t include packages using cdbs’ simple-patchsys, so there are probably really close to 1/4 or 1/3 of our packages using some sort of patch system. How course, having simple-patchsys, dpatch and quilt sounds a bit stupid. But last time I checked, nobody took the time to blog a comparison of those tools, saying why {quilt,dpatch} is better than {simple-patchsys,dpatch,quilt}. Also, simple-patchsys probably makes a good-enough job for its users, so they don’t see the point in changing.

Of course, it’s easier to bury your head in the sand, and ignore the fact that those tools solve real problems for so many people and packages, and make their lives easier. :-)