I’ve spent some more time on Debian Maintainer Dashboard (DMD), a maintainer/team-centric dashboard relying on Ultimate Debian Database.
– added tabs (jquery really rocks) — see example
– added a hacking.html file that describes how to setup a development environment
– created a pad (http://debian.titanpad.com/DMD) to store ideas.
– recruit contributors ;)
The brand new machine for Ultimate Debian Database motivated me to do UDD-related work again, so I implemented an old idea: build a maintainer/team-centric dashboard relying on UDD, the Debian Maintainer Dashboard.
The idea is to expose as much useful information as possible about a maintainer’s packages, both in the traditional (Developers Packages Overview-like) “big table with all the info”, but also in a task-oriented listing (to answer the “ok, I have two hours for Debian, what should I do now?” question).
At this point, the Debian Maintainer Dashboard already brings several cool features (try it with my packages or pkg-ruby’s packages, or your own!):
- Reports buildd issues (failed builds)
- Upstream status monitor that works (UDD has its own)
- Knows about teams’ VCS (if your team uses PET) and display the current version in the VCS
- Lists packages you maintain or co-maintain, and also the package you showed interest for by subscribing to them on the PTS
Of course, as of now, it’s pretty ugly: that’s the “help needed” part of the post. I really suck at web design, and would really appreciate if someone would help me to turn it into something nice to use. Besides adding links and colors everywhere, it could probably be nice to add sortable tables, and to create tabs for “TODO items”, “Versions” and “Bugs” (and possibly Lintian, for example). Contact me if you are interested.
I like to think that archive rebuilds play an important role in Debian Quality Assurance and Release Management efforts. By trying to rebuild every Debian package from source, one can identify packages that do not build anymore due to changes in other packages (compilers, interpreters, libraries, …). It is also a good way to stress-test all packages that are involved in building other packages.
Since 2007, I had been running Debian archive rebuilds on the Grid’5000 testbed, a research infrastructure for performing experiments on distributed systems – HPC/Grid/Cloud/P2P. I filed more than 6000 release-critical bugs in the process.
Late last year, Amazon kindly offered us a grant to allow us to run such QA tests on Amazon Web Services. With Sébastien Badia, we ported the rebuild infrastructure to AWS (scripts), and several rebuilds have already been carried out on AWS.
On the technical level, 50 to 100 EC2 spot instances are started, and then controlled from a master instance using SSH. On build instances, a classic sbuild setup is used. Logs are retrieved to the master node after rebuilds, and build instances are simply shut down when there are no more tasks to process. Several tasks are processed simultaneously on each instance, and when they fail, they are retried again with no other concurrent build on the same instance, to eliminate random failures caused by load or timing issues. All the scripts are designed to support other kind of QA tests, not just rebuilds.
Moving to Amazon Web Services will facilitate sharing the human workload of doing those tests. It is now possible for developers interested in custom tests to do them themselves (hint hint).
I’ve just updated the Debian Packaging Tutorial. This new version addresses a few comments and questions I received over the past months.
Note that there are also french and spanish versions of the tutorial, and I’m of course open to adding other translations.
The tutorial can be found in the packaging-tutorial package (PDF files are in /usr/share/doc/packaging-tutorial/), or on www.debian.org (see links above).
> If Ubuntu 12.04 if a LTS release, and Ruby 1.8.7 goes out of support in June of
> 2013, then why is the default still 1.8.7?
> Ruby 1.9.2 was released in 2010. Ruby 1.9.3 was released in October of this year.
First, there’s almost nobody in the Ubuntu development community doing any Ruby work. Packages are just imported from Debian, and Ubuntu follows what is done on the Debian side.
In the Debian/Ruby team, we are currently transitioning to a new packaging helper (gem2deb) that makes it much
easier to support several Ruby versions. Once this will be done, switching to 1.9.3 by default will be very easy. We already provide a way for the sysadmin to change the default on a system.
Now, doing that transition takes time, and we could have used *your* help (and could still use it). We are still quite on time to do it for Debian wheezy, but it sounds very hard to do it for Ubuntu 12.04 unless someone from Ubuntu steps
up to help.
I must admit that I’ve never been a big fan of the dash as /bin/sh change. I have three main problems with the switch:
POSIX compliance as an argument
Complying to standards is a really good thing. But when everybody is ignoring the standard because they want the comfort of newer features, maybe it’s a sign that the standard should be updated to include those newer features. Most of the bashims used everywhere in scripts were signifiant improvements, like the ability to write:
cp short1/path1/very/long/common/path/to/a/file short2/path2/very/long/common/path/to/a/file
The option to improve bash was not fully explored
We started with the premise that bash is bloated, slow, and cannot be improved. Maybe you can help me with that, but I could only find a few simplistic benchmarks comparing dash and bash, and I could not find any analysis of why bash is slow, and why it cannot be improved.
One of the obvious problems is that bash is also an interactive shell, and is linked to ncurses and terminfo, which increases the startup time. But have we investigated the possibility to have a /bin/bash-non-interactive binary that would not be linked to ncurses?
The change was brought to users
While it is OK for Debian (or Ubuntu, in that case, since that change was done in Ubuntu first) to force its developers to use POSIX-compliant scripts, the switch could have been made only to Debian-created scripts (by switching them from a /bin/sh shebang to a /bin/dash shebang, for example). I have trouble justifying that this change was forced on users as well.
Next: linker changes
… and we are doing it again. A set of linker changes (see also the Ubuntu page) was already done in Ubuntu, and is very likely to be done in Debian as well. This switch requires deep changes in some buildsystems (it requires ordering of libraries and forbids indirect dependencies), and is rather painful (it was reverted before the Ubuntu 11.04 release because it was not possible to fix all the packages during the natty release cycle, but is done in the 11.10 release). Of course, there are justifications for this change. But I’m not sure that it’s worth all the trouble created for users.
Windows has a feature called Dynamic Disks, which makes it ignore the DOS partition table and manage partitions its own ways. gparted is supposed to detect that, and warn the user that it can’t deal with that. The solution is to reboot in Windows, switch the partition back to “Basic disks”, and then proceed with installing Debian (or Ubuntu).
However, a student of mine had an interesting experience because gparted did not detect the dynamic disk stuff and did not warn about it, so the student went ahead with installing Debian squeeze. After reboot in Windows, Windows sees the partition as “invalid”. The same happens with Ubuntu 11.04 (not really surprising).
I’m surprised I did not find much information about that issue. Does someone has more info ?
The student was using two disks in his laptop, and only the second one (not the one used to boot windows) was using “dynamic disks”.
I must admit that I’m a bit lost about the conclusions of the Don’t fear the fsync()! (lf.org down; google cache) debate. My understanding was that using
fsync() was the right thing to do when we cared about data being written to disk.
tar, I usually care about my data being written to the disk, so why don’t they use
fsync()? Shouldn’t they?
So, the fifth Debconf I’ve had the chance to attend is clearly over now. It was great, and all organizers (and sponsors) really deserve huge “thank you” for making this event so successful. I’m already looking forward to next year’s edition.
Debconf has been very productive for me. I chaired 4 sessions:
- the usual Quality Assurance BOF, which was a bit depressing: even if work gets done, the QA “team” doesn’t really feel like a “team”. Maybe that’s because a QA “team” is not needed, and we should instead split it into smaller teams focusing on subsets of the QA work (archive-wide testing, QA services, etc.)?
- a BOF on Ruby, where I demoed the work we have been doing around gem2deb, our new dh-based packaging helper [Ruby team website]
- a tutorial on archive testing, hoping to get more people involved in tests such as archive rebuilds. If you are interested in helping with reporting bugs, please drop me an email [doc about archive tests on wiki.d.o]
- a BOF on finding a ‘standard’ Git workflows for packaging teams. More work is needed on this, but it looks like a good start. [thread on -devel@, wiki page, BOF notes]
I’ve also made numerous uploads of Ruby-related packages, and reduced my backlog on UDD to a reasonable level. I even managed to make a developers-reference upload, integrating all the pending patches from the BTS.
In other news, I’m very excited about the recent progress on expo.debian.net (a mentors.debian.net replacement), which could help streamline our sponsorship process.
Next week, I’ll head to Strasbourg for Rencontres Mondiales du Logiciel Libre 2011. On monday morning, I’ll be giving my Debian Packaging Tutorial for the second time. Let’s hope it goes well and I can recruit some future DDs!
Then, at the end of July, I’ll attend Debconf again. Unfortunately, I won’t be able to participate in Debcamp this year, but I look forward to a full week of talks and exciting discussions. There, I’ll be chairing two sessions about Ruby in Debian and Quality Assurance.