Free software improvements

Two things that have been annoying me for a long time have been fixed recently, and I discovered those two new features the same day (rainy week-ends are good for you).

  • OpenSSH’s problem with networks with large bandwidth-delay product has been fixed in OpenSSH 5.1. The problem was that OpenSSH has its own windowing system over TCP’s, and that its buffer were two small to be able to maximise the bandwidth on, say, a 1 Gbps link with 10ms latency, so you would get a ridiculous bandwidth because of that, even if TCP was correctly tuned. A patch (hpn-ssh) has been existing for a long time, but strangely, people don’t like patching their SSH servers and clients.
    That means I can now get decent bandwidth using scp or rsync over long distance links.
  • The radeon driver now supports TV out for R600 cards (including my X1250). I can now stop using the proprietary fglrx driver, and make the computer under the TV completely free!

OK, both changes have been there for a long time, but I’m sure I’m not the only one who didn’t notice them at the time. Big Thank You to both groups of developers (especially to the ati/radeon developers, who have been very helpful on IRC to troubleshoot my setup).

Bootstrapping Centos or Fedora from Debian or Ubuntu

Here are some notes about bootstrapping a Centos or Fedora chroot from Debian. It should also work from Ubuntu with minor changes, but I haven’t checked. The following should really be done in a chroot, since some commands will install files in your /etc or elsewhere, ignoring the --installroot passed to yum. The following instructions are for Centos, but replacing all occurences of centos with fedora should work.

  • apt-get install yum rpm python-m2crypto. If at some point, you get error messages about rpmlib(BuiltinLuaScripts), you need to install a newer rpm package (from Debian unstable, for example).
  • mkdir -p /tmp/centos/var/lib/rpm
  • rpm --root /tmp/centos --initdb
  • Go to http://rpm.pbone.net or http://www.rpmfind.net, search for centos-release or fedora-release, and download the rpm for the version you want.
  • rpm -ivh --force-debian --nodeps --root /tmp/centos centos-release*rpm (that populates /tmp/centos/etc with information about the centos repositories)
  • yum --installroot /tmp/centos/ install yum . That fails because of missing GPG information in /etc/pki. Do ln -s /tmp/centos/etc/pki /etc/pki, then again yum --installroot /tmp/centos/ install yum.
  • mount -t proc foo /tmp/centos/proc
  • mount -t sysfs foo /tmp/centos/sys
  • chroot /tmp/centos /bin/bash --login

If you get errors about different DB versions between Debian’s RPM and CentOS’ RPM, you can try, in the CentOS chroot:

  • cd /var/lib/rpm && rm * (simplest way to avoid problems between db versions for Debian’s RPM and centos’ RPM)
  • rpm --initdb
  • yum install yum (again, to restore the rpm db)
  • yum install vim-minimal less
  • That’s all!

Update: Jaldhar Vyas pointed me to mach, and Paul Wise to mock. Both packages are available in Debian, but use config files for each release shipped in the package. Unfortunately, both packages are out of date, and don’t include Fedora 9 or newer. Also, mock doesn’t support Centos.
Anyway, both packages could use a new maintainer. Don’t hesitate to jump in!

Managing your PTS subscriptions

Subscribing to packages on the PTS is really useful to stay informed of everything happening with those packages. Unfortunately, managing your PTS subscriptions is really a challenge. Here is how I proceed to subscribe to packages I forgot to subscribe after uploading them:

  • To get the list of your subscriptions, connect to master.d.o, and run
    /org/packages.qa.debian.org/bin/get-summary-subscribers.pl | grep your@email
    That’s also a way to know who is interested in your packages. :-)
  • You can use UDD (from master) to know the list of packages you should probably be subscribed to, that is, packages you maintain or co-maintain in sid:
    psql -A -t service=udd -c "select source from sources
    where distribution='debian' and release='sid'
    and maintainer_email='your@email'
    union select source from uploaders
    where distribution='debian' and release='sid'
    and email='your@email';"
  • You can them use combine to find the packages you should subscribe to, and mass-subscribe using the email interface.
  • Then, you are done, or almost done: mass-confirming PTS subscriptions is a PITA because of #340863.

Some more info:

  • The PTS lives on master, in /org/packages.qa.debian.org.
  • The script that handles emails send to pts@ is bin/control.pl.
  • The spool dir for pending requests is spool/. The files are simple text files.
  • You can send multiple commands per email. Sending them in the subject might be broken. (at least it failed for me)
  • You don't need to take special care of the From for your emails. It's ignored (unless you subscribe without specifying an email address).

Challenging times for Debian

So, the release team announced yesterday that squeeze will freeze in december 2009. This decision was motivated by several things (TTBOMK):

  • while time-based releases are not a good idea for Debian (“We release when it’s ready”), time-based freezes make a lot of sense, and gives every team the opportunity to do its own scheduling
  • freeze in december / release in spring work well (no freeze during debconf, longer nights to hack on Debian)
  • Ubuntu releases LTS versions every two years. Freezing in december 2009 will allow to synchronize with future LTS releases, provide many packages with the same version, and leverage that for support.

Now, this decision raises several questions:

  • will there be attempts to overrule it? Strangely, the discussion at debconf was quite calm. Of course, there are counter-arguments, but nobody has mentioned the willingness to overrule the decision. This has been mentioned by people who are not at Debconf, who might have a different POV.
  • will we manage to freeze in a reasonable state? We will need to rely on fully functional infrastructure: working buildds, short NEW queue, transitions that don’t block stuff for too long, etc.
  • will we manage to leverage collaboration with Ubuntu? Releasing with about the same versions is one thing, but how will we work together while preparing those versions?
  • after the releases (both Ubuntu’s and Debian’s), users will get to choose between two very similar distributions. We need to think about how Debian will differenciate itself from Ubuntu: what should we emphasize? How are we relevant?

Anyway, it seems that we have gone a long way in just a few years. In january 2006, while I was not a DD yet, I prodded Raphael Hertzog about sending his famous “For those who care about their packages in Ubuntu” email, where we described the Ubuntu release process for their first LTS release. The goal was to give interested DDs a chance to take a look at the status of their packages in Ubuntu, so Ubuntu would release with the best possible version. 3.5 years later, we are talking about synchronizing releases.

Debconf TODO List (help needed and welcomed!)

So, here are the things I plan to work on during Debconf. If you are interested in one of those topics, don’t hesitate to talk to me and help (even if you are not at Debconf). I’ve always dreamed of people squashing items on my TODO list.

Ultimate Debian Database:

  • make all the example scripts nicer: requires HTML skills, mostly, and it’s also a good way to get started with UDD, so I would very much like someone else to do that.
  • split email into name and email in the bugs table (based on work by Olivier Berger)
  • Importer for DEHS (Need to talk with Raphael), MIA status, wanna-build (maybe), britney output (maybe).
  • Rewrite bapase using UDD.

Ruby packaging:

  • Work on the packaging of Ruby 1.9.1 together with the interpreter crew.
  • Continue the work on ruby-support and on the new ruby policy, so it becomes usable.

Other stuff:

  • Release a new version of feed2imap and ruby-feedparser
  • Work on developers-reference a bit. Upload a new version.
  • Work on websec a bit. Upload a new version.
  • Work on xmpp4r. Try to integrate some of the ideas of xmpp4r-simple so people stop using a wrapper that uses busy waiting.
  • Work on suuntux a bit. But that looks compromised since I forgot my Suunto serial cable.

Update:

  • UDD importer for DEHS done, after a SQL hacking session with raphael

Slides from RMLL (and much more)

So, I’m back from the Rencontres Mondiales du Logiciel Libre, which took place in Nantes this year. It was great to see all those people from the french Free Software community again, and I look forward to seeing them again next year in Bordeaux (too bad the Toulouse bid wasn’t chosen).

The Debian booth, mainly organized by Xavier Oswald and Aurélien Couderc, with help from Raphaël, Roland and others (but not me!), got a lot of visits, and Debian’s popularity is high in the community (probably because RMLL is mostly for über-geeks, and Debian’s market share is still very high in this sub-community).

I spent quite a lot of time with the Ubuntu-FR crew, which I hadn’t met before. They do an awesome work on getting new people to use Linux (providing great docs and support), and do very well (much better than in the past) at giving a good global picture of the Free Software world (Linux != Ubuntu, other projects do exist and play a very large role in Ubuntu’s success, etc). It’s great to see Free Software’s promotion in France being in such good hands. (Full disclosure: I got a free mug (recycled plastic) with my Ubuntu-FR T-shirt, which might affect my judgement).

I gave two talks, on two topics I wanted to talk about for some time. First one was about the interactions between users, distributions and upstream projects, with a focus on Ubuntu’s development model and relationships with Debian and upstream projects. Second one was about voting methods, and Condorcet in particular. If you attended one of those talks, feedback (good or bad) is welcomed (either in comments or by mail). Slides are also available (in french):

On a more general note, I still don’t understand why the “Mondiales” in RMLL’s title isn’t being dropped or replaced by “Francophones“. Seeing the organization congratulate themselves because 30% of the talks were in english was quite funny, since in most cases, the english part of the talk was “Is there someone not understanding french? no? OK, let’s go on in french.“, and all the announcements were made in french only. Seriously, RMLL is a great (probably the best) french-speaking community event. But it’s not FOSDEM: different goals, different people. Instead of trying (and failing) to make it an international event, it would be much better to focus on making it a better french-speaking event, for example by getting more french-speaking developers to come and talk (you see at least 5 times more french-speaking developers in FOSDEM than in RMLL).

I’m now back in Lyon for two days, before leaving to Montreal Linux Symposium, then coming back to Lyon for three days, then Debconf from 23rd to 31st, and then moving to Nancy, where I will start as an assistant professor in september (a permanent (tenured) position).

LDLC: plus jamais !

Je viens de vivre une expérience intéressante, qui m’a décidé à ne plus jamais commander de matériel chez LDLC. Je voulais acheter un adaptateur USB vers série (DB9) pour pouvoir continuer à utiliser Suuntux avec mon nouveau PC portable, qui n’a pas de port série.

samedi 7 mars: je vais sur LDLC, et commande un adaptateur USB/série. Je prends le moins cher, car je ne compte pas le pousser dans ses derniers retranchements ni en faire une utilisation intensive: ma montre fait du 9600 bauds de toute façon.
mardi 10 mars: le facteur passe une première fois. Mais comme LDLC fait toujours exprès de mettre un carton énorme pour éviter qu’il rentre dans la boite aux lettres (même si votre commande rentrerait largement dans votre boite aux lettres avec un carton adapté), je dois attendre que le facteur passe 2 fois, puis aller chercher le colis à la poste (un plaisir).
jeudi 12 mars: je peux enfin récupérer mon cable. Je branche, le périphérique est correctement détecté par le noyau, mais ça ne marche pas: je ne lis que des 0 (caractère numéro 0, pas caractère ascii “0”). Je cherche pendant une heure, sans réussir à trouver la solution. Ca ne marche pas non plus avec un autre PC. Le périphérique marche bien quand je le branche sur un autre PC avec un port série. Premier mail au support de LDLC.
vendredi 13 mars: premier mail du support, qui me demande plus d’informations sur les tests réalisés.
jeudi 19 mars: après quelques échanges avec le support de LDLC, je renvoie l’adaptateur (deuxième passage à la poste), et j’en recois un autre (troisième passage à la poste). A l’ouverture du colis, je commence à comprendre l’origine de mes problèmes: visiblement, les adaptateurs USB/série vendus par LDLC sont fabriqués à l’arrache: le boitier était mal moulé (tordu), et il manquait même une des vis. Et, ça ne marche toujours pas.
prise1prise2
jeudi 30 mars: je renvoie l’adaptateur USB/série pour recevoir un avoir. (quatrième passage à la poste)
lundi 6 avril: je recois le rapport du service de test de LDLC. Résultat: Cable Usb/Vga OK. !! J’espère qu’ils n’ont pas trop abimé leur prise VGA (3 rangées de pins) en la branchant sur l’adaptateur série (2 rangées de pins).
rapport1
jeudi 9 avril: je recois dans ma boite aux lettres un adaptateur commandé, pour 5 euros de plus, sur Amazon. Je le branche, il marche immédiatement. (d’ailleurs c’était le même driver que ceux de LDLC)
lundi 20 avril: 3 autres échanges de mails pour réussir à récupérer l’avoir. Ils m’avaient oublié.

Conclusion:
– beaucoup de temps perdu pour avoir essayé d’économiser 5 euros
– le service client de LDLC est réactif (rarement plus de deux jours d’attente pour une réponse) mais pas efficace du tout
– LDLC vent du matériel qui ne fonctionne pas
– Le service de test de LDLC ne sait pas reconnaitre une prise série DB9 d’une prise VGA
– Renseignements pris, il semble que la qualité de service de LDLC s’est sérieusement dégradée ces derniers temps (depuis un rachat), et qu’il vaut mieux se tourner vers des concurrents, comme RueDuCommerce ou Materiel.net.

On the New Maintainer process

So, I decided to raise the topic of our New Maintainer process again. To summarize it, our process is the following:

  • Applicant applies, and get advocated by an existing DD. (Takes a few days at most)
  • An Application Manager is assigned to the applicant. Due to the shortage of people willing to help as Application Managers, it often takes several months to get an AM assigned.
  • The Application Manager asks more than 50 questions to the applicant. The questions cover legal stuff, and more technical stuff about packaging and Debian procedures. For the curious, the templates that almost all Application Managers follow are available in the nm SVN repository (see nm_pp*.txt and nm_ts*.txt). Answering all the questions often takes several months, because after each set of questions, the applicant might have to wait for his AM to find the time to review his answers.
  • After all the questions have been properly answered, the AM writes a report (about 1 hour of boring work), that is sent to the NM Front Desk.
  • A member of the NM Front Desk reads the report, and ask additional questions or pass the report to the Debian Account Managers. This step used to take a very long time, but the situation has improved with the addition of new FD members.
  • A member of the DAM read the report, and decides to allow the applicant to become a DD, or reject the applicant. This step still takes several months.

This process has two flaws:

  • It doesn’t help to produce an OS: Debian is not about educating people, it is about producing an operating system. The process of asking questions doesn’t produce anything useful for Debian.
  • It is long and boring, even for good candidates. Many people drop out of the process simply because they get too frustrated of waiting for their AM, FD or DAM. A DD once told me that when people get out of NM and become DD, most of their Debian life has already passed. This is quite true.

So, I proposed to change the process, to something a lot simpler:
When someone wants to become a DD, he seeks advocates amongst the current DDs. Each advocate writes a recommendation email stating under which conditions he worked with the applicant, and why he thinks the applicant should be made DD.
After a sufficient number of advocates have been found (5, for example), the applicant goes through a shorter NM process, only answering 5 or 10 questions, chosen amongst the 50+ from the templates.
Everything is then forwarded to DAM, who takes the final decision.

This proposal received strong opposition. Apparently, people feel that all (or most of) the questions are really necessary, and that relying on the judgement of random other DDs would lower the quality of Debian. Other people consider that it is not a problem if the NM process is taking more than a year for most applicants (even for very good applicants).

I personnally think that we should judge applicants on the skills that will matter when they will be DDs: social skills, technical skills (based on real work, not on their ability to understand documentation), and the consequence of both, the ability to ask for help when there’s something they don’t know. The current NM process mostly measures their ability to use google to search through existing documentation. My proposal didn’t make the process easier, it only made it shorter and less boring: finding 5 advocates is going to be very difficult, because the applicant will have to convince 5 different DDs to say: “I think that X is ready to be a DD now.”

Also, Debian is not the only project trying to recruit new developers, and we are clearly one of the less appealing currently. By not being more open to new contributors (by more open, I mean: shorter and less boring process, not process requiring less skills), we shoot ourselves in the foot. In the end, we reduce the relevance of Debian.

It is quite clear based on the discussion that we won’t get rid of the 50+ questions anytime soon. However, the suggestion to merge FD and DAM (one way or another) emerged from the discussion, which might help with the bottleneck at the end of the process. That would already be a good thing.

Finally, the discussions about the NM process are always interesting, because lots of people have lots of things to say about it, despite never having been an AM. So, to help reading the discussion, here is the full list of Application Managers who had at least one NM that became a DD over the last two years (yes, there are only 41 DDs in that case):

Andreas Barth Mohammed Adnène Trojette Ana Beatriz Guerrero López Anibal Monsalve Salazar
Alexander Sack Bas Zoetekouw Ben Hutchings Bernhard R. Link
Bruno Barrera Bernd Zeimetz Don Armstrong Enrico Zini
Felipe Augusto van de Wiel Alexander Wirt Francois Marier Gunnar Wolf
Marc Brockschmidt Simon Huggins Joerg Jaspert Kari Pahula
Cyril Brulebois Kurt Roeckx Lucas Nussbaum Luk Claes
Pierre Habouzit Martin Wuertele Martin Meredith Michael Koch
Christoph Berg Jonathan McDowell Paul Wise Riku Voipio
Santiago Ruano Rincón Patrick Schoenfeld Reinhard Tartler Martin Michlmayr
Thijs Kinkhorst Thomas Viehmann Steffen Joeris Wouter Verhelst
Martin Zobel-Helas

Recent ssh-agent problems?

I’ve recently expected several ssh-agent problems on my laptop using a mix of Debian testing and unstable.

  • When I use offlineimap to fetch my mail, opening several (up to 5) SSH connections concurrently, sometimes one of the connection asks for a password. When I try again, it just works.
  • Sometimes, the ssh-agent process starts refusing connections, so I’m always asked for the passphrase. The only way to fix that is to restart the ssh-agent, by restarting my GNOME session.

Has someone else experienced those problems? It looks like a fairly recent regression.

UDD and packages metrics

Peter Eisentraut played with Ultimate Debian Database, and wanted to create a “maintenance effort” metric by multiplying each package’s installed size by its popcon. His query is:
SELECT rank() OVER (ORDER BY score DESC), source,
sum(installed_size::numeric * insts) AS score
FROM packages JOIN popcon USING (package)
WHERE distribution = 'debian' AND release = 'sid'
AND component = 'main' AND architecture IN ('all', 'i386')
GROUP BY source, version ORDER BY score DESC LIMIT 30;

Besides all the interesting things that I learnt by looking at his query (rank(), and a bug in UDD because installed_size should really be numeric to avoid the conversion), Peter had a problem with his query: linux-2.6 is missing from the results, while it should obviously have a large popcon and a large install size.

The problem is that the binary packages for linux-2.6 often change, so they don’t get very high in popcon. The unstable kernel package gets a ridiculous popcon score:

select package, insts from popcon
where package in (select package from packages where source ='linux-2.6' and release='sid')
order by insts desc limit 30;

package                           | insts
----------------------------------+-------
linux-libc-dev                    | 38703
linux-source-2.6.29               |   614
linux-headers-2.6.29-2-common     |   256
linux-image-2.6.29-2-amd64        |   239

A solution could be to change the metric to be: MAX(insts over all binary packages from this source package) * SUM(installed_size)
The good thing is that UDD already offers a popcon_src view, that gives the popcon score for a source package. So the query becomes:
SELECT rank() OVER (ORDER BY score DESC), source,
sum(installed_size::numeric * insts) AS score
FROM packages JOIN popcon_src USING (source)
WHERE distribution = 'debian' AND release = 'sid'
AND component = 'main' AND architecture IN ('all', 'i386')
GROUP BY source ORDER BY score DESC LIMIT 30;

 rank |     source    |    score
------+---------------+-------------
1 | openoffice.org    | 92177633504
2 | qt4-x11           | 18503941620
3 | linux-2.6         | 16036201020
4 | gcc-4.3           | 14369300376
5 | mesa              | 12962475968
6 | eglibc            | 12581290240
7 | gcc-4.4           | 11411296672
8 | samba             | 10021083072
9 | xulrunner         |  9037295424
10 | mysql-dfsg-5.0   |  8348333532

This time, linux-2.6 shows up near the top of the list.