Half of the package maintainers are not DDs or DMs

November 26th, 2012 by lucas

During the Paris Mini-Debconf, Nicolas Dandrimont talked about The state of mentors.debian.net: GSoC and beyond. He said that Half of Debian’s packages are maintained by sponsored maintainers. That statement was actually wrong, as he confirmed later.

However, using a few UDD queries, I could come up with:

  • 3147 packages out of 18649 packages in sid (17%) were last uploaded using a sponsored upload.
  • There are 963 distinct sponsorees, vs. a total of 2015 distinct emails in the Changed-By field of packages in sid. Given that it’s more likely for DDs to have used several emails to upload packages, it’s very likely that half of the package maintainers are sponsored maintainers.
  • There are 2185 packages without a DD in either the Maintainer: or the Uploaders: fields. (That includes some packages that are maintained by a team that could include some DDs)

Full UDD notes:

all packages in sid:
select source, version from sources_uniq where release = 'sid'

packages in sid known to upload_history:
select source, version from upload_history where
(source, version) in (select source, version from sources_uniq where release = 'sid')

packages that were uploaded by the changed_by person:
create temporary table sources_not_sponsored as select distinct source, version
 from upload_history, carnivore_keys, carnivore_emails
 where (source, version) in (select source, version from sources_uniq where release = 'sid')
 and fingerprint = key
 and carnivore_keys.id = carnivore_emails.id
 and carnivore_emails.email = changed_by_email;

packages not uploaded by the changed_by person:
create temp table uh_sid as select source, version, fingerprint, changed_by_email
from upload_history
where (source, version) in (select source, version from sources_uniq where release = 'sid');

create temp table uh_sid_sponsored as select source, version, fingerprint, changed_by_email from uh_sid
where (source, version) not in (select source, version from sources_not_sponsored);

list with sponsor login:
select distinct source, version, fingerprint, changed_by_email, login
from uh_sid_sponsored
left join carnivore_keys on fingerprint = key
left join carnivore_login on carnivore_keys.id = carnivore_login.id;

=> 4188 sponsored packages. some of them are in a strange state (changed_by is a DD, but uploaded by another DD). excluding those:

create temp table sponsored_but_dds as select distinct source, version, fingerprint, changed_by_email, login
from uh_sid_sponsored, carnivore_emails, carnivore_login
where changed_by_email = carnivore_emails.email
and carnivore_emails.id = carnivore_login.id;

create temp table really_sponsored as select distinct source, version, fingerprint, changed_by_email, login
from uh_sid_sponsored
left join carnivore_keys on fingerprint = key
left join carnivore_login on carnivore_keys.id = carnivore_login.id
where (source, version) not in (select source, version from sponsored_but_dds);

=> 3147 sponsored packages

select distinct changed_by_email from really_sponsored ;
=> 963 different sponsorees

select distinct changed_by_email from upload_history where
(source, version) in (select source, version from sources_uniq where release = 'sid');
=> 2015 distinct emails.

no DD amongst maintainer or uploader:

create temp table dds_emails as select email from carnivore_emails, carnivore_login
where carnivore_emails.id = carnivore_login.id;

select source, version, maintainer, uploaders from sources_uniq
where release='sid'
and maintainer_email not in (select * from dds_emails)
and not exists (select * from uploaders where release = 'sid' and sources_uniq.source = uploaders.source and sources_uniq.version = uploaders.version and email in (select * from dds_emails))
and maintainer_email != 'packages@qa.debian.org'
and (source, version) in (select source, version from really_sponsored);

Mini-debconf lightning talk: Get involved! It’s not that hard!

November 26th, 2012 by lucas

This week-end I attended the Paris Mini-Debconf, which was really a great event, and a nice opportunity to meet everybody again.
I delivered a lightning talk on “Get involved! It’s not that hard!“, which was also a good excuse to mention the Debian packaging tutorial and the Debian Maintainer Dashboard.

Release Critical Bugs report for week 43

October 25th, 2012 by lucas

During the Squeeze release cycle, Alexander Reichle-Schmehl used to blog every week about this. Let’s try to continue the tradition.

The UDD bugs interface currently knows about the following release critical bugs:

  • In Total: 916
    • Affecting wheezy: 464
      That’s the number we need to get down to zero before the release. They can be split in two big categories:

      • Affecting wheezy and unstable: 327
        Those need someone to find a fix, or to finish the work to upload a fix to unstable:

        • 77 bugs are tagged ‘patch’.
          Please help by reviewing the patches, and (if you are a DD) by uploading them.
        • 17 bugs are marked as done, but still affect unstable.
          This can happen due to missing builds on some architectures, for example. Help investigate!
        • 235 bugs are neither tagged patch, nor marked done.
          Help make a first step towards resolution!
      • Affecting wheezy only: 137
        Those are already fixed in unstable, but the fix still needs to migrate to wheezy. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.

Note: I’m looking for someone to send this posts on a regular basis. All you need is a blog syndicated on Planet Debian (there’s a secret UDD page that generates the HTML code for you ;) ). Contact me if you are interested. taken!

Debian Maintainer Dashboard update

July 6th, 2012 by lucas

I’ve spent some more time on Debian Maintainer Dashboard (DMD), a maintainer/team-centric dashboard relying on Ultimate Debian Database.

Done:
- added tabs (jquery really rocks) — see example
- added a hacking.html file that describes how to setup a development environment
- created a pad (http://debian.titanpad.com/DMD) to store ideas.

TODO:
- recruit contributors ;)

Introducing the Debian Maintainer Dashboard — help needed!

July 2nd, 2012 by lucas

The brand new machine for Ultimate Debian Database motivated me to do UDD-related work again, so I implemented an old idea: build a maintainer/team-centric dashboard relying on UDD, the Debian Maintainer Dashboard.

The idea is to expose as much useful information as possible about a maintainer’s packages, both in the traditional (Developers Packages Overview-like) “big table with all the info”, but also in a task-oriented listing (to answer the “ok, I have two hours for Debian, what should I do now?” question).

At this point, the Debian Maintainer Dashboard already brings several cool features (try it with my packages or pkg-ruby’s packages, or your own!):

  • Reports buildd issues (failed builds)
  • Upstream status monitor that works (UDD has its own)
  • Knows about teams’ VCS (if your team uses PET) and display the current version in the VCS
  • Lists packages you maintain or co-maintain, and also the package you showed interest for by subscribing to them on the PTS

Of course, as of now, it’s pretty ugly: that’s the “help needed” part of the post. I really suck at web design, and would really appreciate if someone would help me to turn it into something nice to use. Besides adding links and colors everywhere, it could probably be nice to add sortable tables, and to create tabs for “TODO items”, “Versions” and “Bugs” (and possibly Lintian, for example). Contact me if you are interested.

Debian archive rebuilds on Amazon Web Services

May 30th, 2012 by lucas

I like to think that archive rebuilds play an important role in Debian Quality Assurance and Release Management efforts. By trying to rebuild every Debian package from source, one can identify packages that do not build anymore due to changes in other packages (compilers, interpreters, libraries, …). It is also a good way to stress-test all packages that are involved in building other packages.

Since 2007, I had been running Debian archive rebuilds on the Grid’5000 testbed, a research infrastructure for performing experiments on distributed systems – HPC/Grid/Cloud/P2P. I filed more than 6000 release-critical bugs in the process.

Late last year, Amazon kindly offered us a grant to allow us to run such QA tests on Amazon Web Services. With Sébastien Badia, we ported the rebuild infrastructure to AWS (scripts), and several rebuilds have already been carried out on AWS.

On the technical level, 50 to 100 EC2 spot instances are started, and then controlled from a master instance using SSH. On build instances, a classic sbuild setup is used. Logs are retrieved to the master node after rebuilds, and build instances are simply shut down when there are no more tasks to process. Several tasks are processed simultaneously on each instance, and when they fail, they are retried again with no other concurrent build on the same instance, to eliminate random failures caused by load or timing issues. All the scripts are designed to support other kind of QA tests, not just rebuilds.

Moving to Amazon Web Services will facilitate sharing the human workload of doing those tests. It is now possible for developers interested in custom tests to do them themselves (hint hint).

Update of Debian Packaging Tutorial

April 13th, 2012 by lucas

I’ve just updated the Debian Packaging Tutorial. This new version addresses a few comments and questions I received over the past months.

Note that there are also french and spanish versions of the tutorial, and I’m of course open to adding other translations.

The tutorial can be found in the packaging-tutorial package (PDF files are in /usr/share/doc/packaging-tutorial/), or on www.debian.org (see links above).

Re: Ubuntu vs Ruby

January 1st, 2012 by lucas

(original post)

> If Ubuntu 12.04 if a LTS release, and Ruby 1.8.7 goes out of support in June of
> 2013, then why is the default still 1.8.7?
>
> Ruby 1.9.2 was released in 2010. Ruby 1.9.3 was released in October of this year.

First, there’s almost nobody in the Ubuntu development community doing any Ruby work. Packages are just imported from Debian, and Ubuntu follows what is done on the Debian side.

In the Debian/Ruby team, we are currently transitioning to a new packaging helper (gem2deb) that makes it much
easier to support several Ruby versions. Once this will be done, switching to 1.9.3 by default will be very easy. We already provide a way for the sysadmin to change the default on a system.

Now, doing that transition takes time, and we could have used *your* help (and could still use it). We are still quite on time to do it for Debian wheezy, but it sounds very hard to do it for Ubuntu 12.04 unless someone from Ubuntu steps
up to help.

dash as /bin/sh, and now ld –as-needed. Pattern?

November 6th, 2011 by lucas

I must admit that I’ve never been a big fan of the dash as /bin/sh change. I have three main problems with the switch:

POSIX compliance as an argument

Complying to standards is a really good thing. But when everybody is ignoring the standard because they want the comfort of newer features, maybe it’s a sign that the standard should be updated to include those newer features. Most of the bashims used everywhere in scripts were signifiant improvements, like the ability to write:
cp {short1/path1,short2/path2}/very/long/common/path/to/a/file
instead of:
cp short1/path1/very/long/common/path/to/a/file short2/path2/very/long/common/path/to/a/file

The option to improve bash was not fully explored

We started with the premise that bash is bloated, slow, and cannot be improved. Maybe you can help me with that, but I could only find a few simplistic benchmarks comparing dash and bash, and I could not find any analysis of why bash is slow, and why it cannot be improved.
One of the obvious problems is that bash is also an interactive shell, and is linked to ncurses and terminfo, which increases the startup time. But have we investigated the possibility to have a /bin/bash-non-interactive binary that would not be linked to ncurses?

The change was brought to users

While it is OK for Debian (or Ubuntu, in that case, since that change was done in Ubuntu first) to force its developers to use POSIX-compliant scripts, the switch could have been made only to Debian-created scripts (by switching them from a /bin/sh shebang to a /bin/dash shebang, for example). I have trouble justifying that this change was forced on users as well.

Next: linker changes

… and we are doing it again. A set of linker changes (see also the Ubuntu page) was already done in Ubuntu, and is very likely to be done in Debian as well. This switch requires deep changes in some buildsystems (it requires ordering of libraries and forbids indirect dependencies), and is rather painful (it was reverted before the Ubuntu 11.04 release because it was not possible to fix all the packages during the natty release cycle, but is done in the 11.10 release). Of course, there are justifications for this change. But I’m not sure that it’s worth all the trouble created for users.

Windows dynamic disks and debian installer ?

September 22nd, 2011 by lucas

Windows has a feature called Dynamic Disks, which makes it ignore the DOS partition table and manage partitions its own ways. gparted is supposed to detect that, and warn the user that it can’t deal with that. The solution is to reboot in Windows, switch the partition back to “Basic disks”, and then proceed with installing Debian (or Ubuntu).

However, a student of mine had an interesting experience because gparted did not detect the dynamic disk stuff and did not warn about it, so the student went ahead with installing Debian squeeze. After reboot in Windows, Windows sees the partition as “invalid”. The same happens with Ubuntu 11.04 (not really surprising).

I’m surprised I did not find much information about that issue. Does someone has more info ?

The student was using two disks in his laptop, and only the second one (not the one used to boot windows) was using “dynamic disks”.