The Project membership procedures GR

So, the call for vote on the “Project membership procedures” GR was sent. Since I took part in the discussions that lead to that GR, I feel obliged to explain my point of view.

What this GR isn’t about
This GR isn’t about the merits of Joerg Jaspert’s proposal. I personally don’t think that the changes he wants to implement are what we need in Debian, but even if I fully agreed with those changes, I would still disagree with the process that led to those changes. Also, this GR isn’t about Joerg Jaspert himself. I deeply respect him and his awesome work for Debian. And I’m sure that this vote could have been triggered by decisions announced by someone else.

What this GR is about
I think that the question we are asked to vote on is:

Some people in Debian have a special role, and special privileges. Does it allow them to change important aspects of our project, without the approval of all the other developers?

Or, put differently:

Where is our limit between Doocracy and Democracy?

I’m all for listening to developers who contribute(d) a lot. I think that most people, when skimming through posts in a big thread on -devel@, will look for posts from people who did a lot of work for Debian, or who usually post interesting ideas. However, that doesn’t mean that those people should feel empowered to decide changes to important parts of our project, without first discussing those changes with all other developers, and making sure that those changes are consensual.

So, I hope that we will send a clear message that says: you are very welcome to propose changes, but please make sure that they are properly discussed within the developers before they are implemented.

Joerg’s initial mail on d-d-a, with a few s/decision/proposal/, and a final question phrased like “Any comments? Suggestions? Other ideas?”, could have been the start of an interesting discussion about how we can integrate non-developing contributors in Debian. And if too many developers disagreed, we could have had an opinion poll (it would be great to have an infrastructure for running polls inside Debian, like what Jeroen did three times a long time ago), or even a GR, like we did for Debian Maintainers.

For this actual vote, there are two similar options on the ballot (choices 1 and 2, respectively “amendment text A” and the original text on the vote page). I proposed this amendement (choice 1) because the original text is unclear, and mixes several different questions, forcing voters to answer several (secondary) questions at once, like:

  • Do we recognize that there are contributors that can’t become members of the project currently, and that should?
  • Do we want to thank DAM for bringing this up? (implicit question or not: … now that we are trying to release?)
  • Do we want DAM to work on a new proposal?

The amendment aims at going straight to the point. And yes, it is only asking the DAMs, but if the developers ask the DAMs not to do something, and the DAMs do it anyway, we clearly have a bigger problem. I believe that the way those decisions were announced was a mistake (that unfortunately none of the people involved have been willing to admit), not that the DAMs are trying to take-over Debian, so there’s no need for a stronger wording.

Developer status: problems and solutions

For a minute, I’d like to think about The Announcement in terms of problems and solutions.

Problem: We don’t have enough non-developing contributors in Debian
Solution in The Announcement: Give them special, official, statuses in the Debian community, so they are rewarded for their work.
Alternate solution: Are they really looking forward this status? Aren’t we just thinking that they are just as power-hungry as we are? Last time I checked, the Debian community wasn’t really welcoming non-developers. It isn’t a problem of official status. We have to work on ourselves to make Debian a better place to be for non-developing contributors. Also, giving those people a second-class citizenship, that won’t be widely recognized — “Debian Member” instead of “Debian Developer”, isn’t going to help. (or even worse, “Debian Contributor”, which doesn’t give any right except an email address).

Problem: We have a trust and security problem. It’s difficult to give upload rights and access to Debian machines to more than 1000 people, with some of them not being active anymore, or not having much experience with security (in the case of non-developing contributors).
Solution in The Announcement: Remove rights based on classes of developers, so the group of people having full access stays within manageable boundaries.
Alternate solution: Have fine-grained control on who can do want.

Problem: DMs can’t get access to Debian resources because their keys aren’t managed by the Debian keyring team, but by another team.
Solution in The Announcement: Keyring managers will take care of the Debian Maintainers keyring too. In the past, the keyring managers were the cause for huge delays in the NM process. It has improved a lot recently, but it doesn’t mean that it will always be like that.
Alternate solution: Make the DM keyring team, the keyring managers, and DSA, work together on a solution. it doesn’t sound impossible. If that’s necessary, merge the DM keyring team and the keyring managers.

Problem: DM was done without the blessing of most of the members of the loosely defined group of powerful DDs (sometimes described as “cabal”).
Solution in The Announcement: Drop it, replace it with something very similar, but originating from the cabal.
Alternate solution: Why not keep it?

Problem: Clueless DDs advocate clueless contributors for DMs. Then those DMs upload crap to the archive. (Note: I don’t necessarily agree with that problem, that’s just a problem that the announcement tries to solve)
Solution in The Announcement: Require answering a few questions before becoming a DM. Remove the right for every DD to decide which DM can upload which package. Give that control only to the NM committee.
Alternate solution: If we can’t trust one DD to take the good decision about advocating someone for DM, we could require that each person needs to be advocated by two different DDs. Or three, four, five. Asking the DM to answer a few questions, whose answers he can probably get by asking around on IRC, is not going to improve the average level of our DMs or DDs.


I believe that there are real problems that need to be solved, but that the decisions announced don’t solve the real underlying problem. Here is what I’d like to see:

  • Fine-grained access control for each DD or DM or …:
    This would allow people to have specific rights inside Debian, like:

    • Right to login on each debian.org host (a per-host switch)
    • Right to upload packages for which the person is Maintainer or Uploader
    • Right to upload any package
    • Right to vote

    All rights would default to “NO”. DDs would be allowed to change their rights for everything, while DMs would only be allowed to change specific rights. Of course, we need a secure way to change rights (HTTPS+confirmation using GPG, like the current “sudo password” thing?). But that would help with the security problem (you wouldn’t have access to all Debian hosts without enabling it). In fact, we already have that: many hosts require a switch to be enabled in LDAP before you can login, and we have tons of Unix groups to restrict access to specific areas of our infrastructure. I’m just proposing to extend that to basically everything, and to allow people to grant themselves some rights without going through DSA.

  • Modified NM process for non-developing contributors, that would still include many P&P and T&S questions, but would give “Debian Developer” status without any upload rights. That way, non-developing contributor would have an official status, and will still be able to write “Debian Developer” in their resume. Giving a different status to those people just because we don’t think that they should have the right to upload any package doesn’t feel right.

Is that enough? Probably not. We need to be more welcoming towards non-developing contributors, which is a social problem, not a technical one. But we are not going to solve that in a GR.

New Debian Developers!

We got a lot of (>= 10) new Debian developers recently. I’m really happy to see that the bottlenecks in the New Maintainer process were (at least partially) solved. My first NM (actually my second, my first one is on hold) also became a DD today.

So, how long does it take to become a DD ? Let’s take 2 examples. Both are very active and skilled new contributors, that probably were quite close from being the faster you can be through NM:

Name Applied AM assigned Approved by AM Account created
Chris Lamb 2008-05-01 2008-06-12 2008-07-22 2008-09-16
Sandro Tosi 2008-03-24 2008-05-06 2008-06-22 2008-09-16

We have the proof: provided you have all the required skills, you can become a DD in less than 6 months!

Of course, some things are not perfect yet:

  • A lot of very good contributors are waiting for an AM, because not enough DDs volunteer to be AMs.
  • Some NMs still take too long to answer questions, using AMs that could probably mentor faster NMs. If your AM is waiting for you, feel guilty now!
  • Front Desk and DAM are still managed by a small set of very active (and very busy elsewhere) DDs. Many of the new DDs were FD-approved and DAM-approved by the same person, which is not so great if we want to keep this two-steps check.

UDD and Buildstat

Ultimate Debian Database (UDD) is a GSoC project (now finished) which aimed at importing all different data sources that we have in Debian in a single SQL database, to make it easy to combine that data. Currently, we import information about source and binary packages, all bugs (both archived and unarchived), lintian, carnivore, popcon, history of uploads, history of migrations to testing, and orphaned packages. The goal is really to have that data, without really thinking of a specific use cases: there will be lots of use cases.

Buildstat is a project by Gonéri Le Bouder, that provides a framework for running QA tests (rebuilds and lintian currently, but buildstat is built in a very extensible way) on packages, using both packages in the archive, and packages in the VCS repositories of teams. This is pretty cool: it allows teams to get an overview of the status of their packages, not using the archive as reference, but using their VCS. Buildstat schedules and runs the tests, store the data in an SQL database, and allows to browse the data using a web interface. Buildstat also import some data from other sources (only the BTS currently, using the LDAP dump) to display it on the web interface.

Since both projects are using an SQL DB, people have been asking why we don’t simply merge them. The big advantage would be that the data is synchronized: buildstat would display more up-to-date info about the data sources it doesn’t generate locally (like bugs), and UDD would get fresh buildstat data. We have been talking a lot with Gonéri, considering the different possibilities. But I don’t think it’s a good idea.

I think that both projects should try to do one thing, but do it very well, instead of trying to fix the world. UDD focuses on importing data that exists elsewhere. Sometimes it means doing some complex processing. But data should be be generated by UDD. Merging the projects would mean having a very big piece of software that does everything (or tries to do everything).

Both databases were designed differently, with different goals. UDD tries to stay close to the data it imports. There might be some incoherences in the data sources, but that’s fine: one of the goal of UDD is to make it easy to find them (and fix them), so we need them in the DB. In buildstat, since the goal of importing data is to display it on the web interface, with a strict use case, you can freely “simplify” data if it helps. Another big difference is that UDD is designed to be easy to use (ie write and run queries) by a human user: UDD uses multi-column primary keys, while buildstat uses surrogate keys (integer “id” keys), that ORM tools usually require.

There are also more technical concerns: currently UDD makes a compromise, for each data source, on what it imports: it tries to import the data that is useful, not all the data available. Merging buildstat and UDD would mean increasing the DB size significantly, by adding all the “private” data that buildstat needs. Another problem is the stable API problem: if buildstat and UDD are merged, it means that buildstat cannot change its DB schema without making sure that it wouldn’t break what UDD users are doing.

So, what should we do, from my POV, instead of merging?

– Continue to talk, and get Gonéri into the UDD “team”. He gathered a lot of experience working on buildstat, and he probably would be able to help a lot.

– Data that is not generated by buildstat (bugs data) should be imported from UDD. Doing SQL->SQL will probably make things easier there.

– A summary of buildstat’s infos should be imported into UDD.

There’s also the issue of providing the data through a web interface, to the DDs, which buildstat tries to address partially. The Debian Developer Packages Overview (DDPO)’s main limitations are:

– lack of knowledge about VCS (what buildstat solves)

– lack of knowledge about complex organizations (you can’t get any list of packages, or list of packages maintained by teams not using a consistant Maintainer/Uploaders scheme, or list of packages from tasks, etc)

– poor handling of large amount of packages. In teams with lots of packages, it’s useful to have restricted views, such as “packages outdated compared to upstream”, “packages which have bugs/RC bugs”, “packages which are newer in the VCS than in the archive”. The perl team’s work on PET clearly shows the kind of things that are needed.

At this point, I think that DDPO would benefit from a full rewrite (using the existing code as a source of inspiration, and making sure that there are no regressions, of course).

Using UDD as the data source, it should be easy to get something done (even if UDD still lacks some of the data DDPO has currently). But it still requires web developer skills, which I don’t have. If you are interested, contact me!

Of popular packages removed from testing, and the Ultimate Debian Database GSOC project

Some time ago, there was some flamewars^H^Hdebate about the Release Team’s removals of RC-buggy packages from testing. Basically, some people claimed that popular packages shouldn’t be removed, even if RC-buggy.

But, do we really miss popular packages in testing?

It’s difficult to know. You could get the popcon data, and compare it with the Packages files for testing and unstable. Or work with source packages (which removes a lot of noise), but then, you have to convert the popcon data (which uses binary packages names) to source packages. Not completely trivial.

That’s where the Ultimate Debian Database GSOC project comes to the rescue. The goal of Christian von Essen’s project is to gather data from various sources in Debian into a single SQL DB, so queries that combine all those data sources can easily be written.

For example, here is the query that lists the source packages that are in unstable, but not in testing, sorted by their popcon (using the number of insts of the most popular binary package of the source package as value for the source package):

SELECT DISTINCT unstable.package, insts
FROM (SELECT DISTINCT package FROM sources
WHERE distribution = 'debian' and release = 'sid') AS unstable, popcon_src
WHERE unstable.package NOT IN (
   SELECT package FROM sources
   WHERE distribution = 'debian' AND release = 'lenny')
AND popcon_src.source = unstable.package ORDER BY insts DESC;

And the results are available on the web!

Top packages (> 1000 insts):

lzo	64962
gnome-cups-manager	32346
db4.6	20708
ffmpeg-debian	12908
freetype1	10569
flashplugin-nonfree	7116
perlftlib	6769
nvidia-graphics-drivers	3864
wxwindows2.4	3640
dvi2tty	2239
kdebase-runtime	1725
easytag	1717
g-wrap	1582
yaird	1507
slocate	1499
youtube-dl	1390
hugin	1275
w3c-libwww	1058

Interested in UDD? Join #debian-qa or debian-qa@lists.d.o (or talk to me @DebConf!)

La France, terre d’accueil

Un graphique intéressant, trouvé sur le blog de Jean-Marc Manach (le journaliste qui a interviewé Samuel Hocevar aux RMLL pour Le Monde):

Je sais pas vous, mais je trouve ça assez triste, moi, tous ces gens prêts à énormément de sacrifices pour venir dans notre pays, qu’on rejette comme de la merde.

A propos, avez-vous lu le rapport de la CIMADE sur la régularisation des familles étrangères d’enfants scolarisés ? Après l’avoir lu, on se rend mieux compte de la chance qu’on a d’avoir le droit d’être en France…

J’ai une idée qui traine depuis très longtemps. Il s’agirait de créer un site mettant en relation des informaticiens et des associations, afin de permettre aux associations de trouver facilement des informaticiens prêts à donner un coup de main bénévolement (pour réaliser des sites web, administrer quelques machines, etc). Mais je ne sais pas s’il y a toujours un besoin: le besoin existait probablement il y a 3 ou 4 ans, mais maintenant, ça doit être beaucoup plus facile de trouver quelqu’un.

Working evince and poppler in Debian Etch

Evince and poppler (the library used to render PDFs) are quite outdated in etch, and fail to display correctly a number of PDF files (including PDFs generated by latex-beamer). There’s a new poppler version (0.5.x, vs 0.4.5 in etch), but it bumps the soname, so it’s a no-go.

Since I was particularly annoyed by this (I’m an evince fan), I looked at the issue, and found that entry in poppler’s changelog for version 0.5.1:

2006-02-16  Albert Astals Cid

        * qt4/src/Makefile.am:
        * qt/Makefile.am:
        * poppler/Makefile.am:
        * glib/Makefile.am: Update soname as we are not really compatible
        anymore with previous releases that had soname 0.0.0

And I thought that maybe, the soname update was not totally justified. I modified the package in experimental (v.0.5.4) to remove the soname bump, and renamed the binary packages so it becomes a drop-in replace for the version in etch.

It works fine for me with the version of evince in etch, and seems to fix at least #369164, #409758, and #410085.

Packages are available here. Of course, this is totally unsupported, and I have not done any careful testing. But it works for me until now, and I’l update this entry if I discover some problems. Also, I don’t plan to maintain this on the long term, so beware of security bugs.

Update : This breaks tetex (see #356079). Arg. I’ll have to use the experimental version…

Debian Package of the Day is now alive again, and needs your help!

For those who have missed the previous blog entries: Debian Package of the Day (aka Deb-a-day) is now alive again ! There has already been several entries (published 2 times a week, on wednesdays and sundays, until we make sure that we can sustain an higher rate):

Deb-a-day is not syndicated on Planet Debian (because of the “only personal blogs” rule), but it’s syndicated on Planet Ubuntu, and it might be syndicated on Debian Times (negociations are still ongoing :-). Don’t forget to subscribe to the feed !

We (the nice team of editors) have 2 entries ready in the queue, but that’s not enough. If you discovered an interesting package through Deb-a-day, and didn’t submit an entry yet, you should really feel guilty now ! (we are especially looking for cool graphical apps, so Deb-a-day becomes less nerdy, but apps for nerds are welcomed as well, of course.)

Of Debian Etch’s quality and release schedule

The delay of Debian Etch caught some bad press this week. Some articles compare Debian to Ubuntu (which tries hard to have fixed-time releases), and some bloggers are amused by the fact that the next Debian release was delayed “again”. There are some important points though:

  • Etch wasn’t really scheduled to be released on Dec 4th. This target date was mainly used to determine other dates in the release process, like the freeze dates.
  • Ubuntu Dapper was delayed as well, for 6 weeks. Ubuntu Edgy wasn’t delayed, but I don’t think anybody can seriously compare Ubuntu Edgy’s quality with the upcoming Debian etch’s quality: edgy was more like a technology preview.

Now, the question is: how good is Debian Etch compared to Ubuntu Dapper ? It’s difficult to compare distributions: there aren’t a lot of good metrics for this. Of course, one could compare the number of packages that fail to build from source (it’s a good indicator of something seriously wrong in a package). But a lot of work has been done on etch about this, so it wouldn’t really be fair for Ubuntu. A good alternative is debcheck.

debcheck checks that packages can be installed (i.e that their dependancies can be satisfied). It does so by analyzing the content of the Packages files. I compared the results of debcheck on etch as of today (I re-processed it, but the same results are available from qa.d.o), and of debcheck on Dapper (not including dapper-updates – I wanted to compare quality at release date). Included sections were main for etch, and main, restricted and universe for dapper (I didn’t want to consider non-free packages, since their quality tend to be lower).

Results:

4 packages have unsatisfiable Depends on etch, on i386. 49 have unsatisfiable Recommends.

In the main section of Dapper, still on i386, only one package has a problem with its Depends (it was psycopg, because its binary package python2.3-psycopg has a dependancy on python2.3-egenix-mxdatetime). 46 packages in main had unsatisfiable Recommends. This is better than Etch, of course, but Ubuntu/main is much smaller than Debian/main.

When restricted and universe are included, the results are much worse. 80 packages have unsatisfiable Depends, and 150 packages have unsatisfiable Recommends. It is worth noting that those numbers are worse than those of Debian unstable as of today (67 packages have unsat Depends, 34 have unsat Recommends).

Conclusions (sort of):

  • Using the debcheck metric (which is actually quite important, since it translates in uninstallable packages for the users), Debian etch is already of better quality than Ubuntu Dapper when it was released (again, I haven’t checked with dapper-updates included).
  • It would be great to integrate such tests into the Ubuntu release process. Fixing bug is good, but such tools help to improve the distribution quality as a whole. I’ll try to work on this for Feisty after the Etch release.

Features vs. Freedom

Jono Bacon wrote a long blog entry on Planet Ubuntu about his vision of freedom, and how it applies to the proprietary drivers. This is a good opportunity to write sthing I wanted to write for a long time.

Ubuntu’s bug #1 is “Microsoft has a majority market share“. Well, I think that this should be of severity minor or wishlist, not critical. A better bug #1 would be “our priority is our users” (does it ring a bell ?). I find it far more important to improve the satisfaction of people already using a Linux distribution, than to try to convince others to use it.

Most users of Windows have very good reasons for using Windows, like proprietary applications that have no equivalent in the Free Software world. I don’t think that “Linux doesn’t have a 3D desktop” is the major blocker for people not using Linux. Windows doesn’t have a 3D desktop. Many Mac OS X users don’t use the 3D features. I haven’t felt the slightest need to try a 3D desktop, and I’m using GNOME/metacity, so the jump wouldn’t so hard to make to compiz.

I’m not saying that a 3D desktop is not a good idea. Of course, it would be nice to have LiveCDs that just work and start a 3D desktop, so we could show off at conferences. But I haven’t even bothered to google for such LiveCDs.

Sometimes, proprietary drivers are more needed on Linux, for example for people who bought a Wifi card without checking first if it was supported. But I don’t think that 3D graphic drivers are that important. I still hope that Ubuntu will not ship proprietary software by default, and that Ubuntu will try to help the free drivers instead.

Update: I forgot to close the comments when publishing this entry. They are closed now, but trackbacks are open if you want to write your own blog about this (I don’t think it’s necessary, all arguments have been reharsed many times already. Ah, and for the records, the free ati driver works perfectly fine and fast (using MergedFB) in my dual-screen setup (2560*1024), with xv on both monitors.