Debian is changing

I’ve been playing with snapshot.d.o, which is a fantastic resource to look at how Debian has been changing over the years.
Below are some preliminary results about various aspects of source packages: format, packaging helpers, and patch systems.

For clarification:

  • “modif-files” indicates packages that do not use a patch system, but modify files outside debian/.
  • “no-modif-files” indicates packages that do not use a patch system, but do not modify files outside debian/.
  • “other” is “everything else”. Like packages using dpatch, but still modifying some files outside Debian. Or using both dpatch and quilt. (yes, some packages do that).

Random comments:

  • People have been moving away from simple-patchsys and dpatch to quilt since 01/2009 (their usage has been decreasing since then). It seems that discussing the introduction of the new package formats was enough to make people move away from dpatch, even if the 3.0 (quilt) was not usable yet. We now have less than 1000 packages still using dpatch or simple-patchsys. Can we get rid of them all for wheezy? It would be nice to standardize on quilt.
  • The growth of the 3.0 (quilt) format is impressive. Its adoption is much faster than the adoption of dh7. That surprises me a lot, since I had the impression that more people were unhappy about 3.0 (quilt) than about dh7.
  • The number of packages with no packaging helper, but no modified files outside debian/ is dropping. Either than means that we are patching upstream more and more (unlikely), or that means that people are switching to 3.0 (quilt) even if they have no patches, probably due to the other benefits! Could this mean that we could standardize on 3.0 (quilt) and 3.0 (native) for wheezy or wheezy+1?
  • The number of packages that modify files outside debian/ without using a patch system has been constantly dropping since 2005.
  • dh7 is not a cdbs killer. While it seems to be the cause for a huge drop in “classic debhelper” packaging, the number of packages using cdbs is still progressing.

If you have other ideas of things to investigate using snapshot.d.o, don’t hesitate to ask in comments.

Introducing the Debian packaging tutorial

One of the common complains about Debian packaging is that it’s hard to learn because, while there is quite a lot of high-quality documentation, it is often written more as a reference manual than as a tutorial: it’s great if you already know everything and want to check some detail, but not so great if you want to learn everything from scratch.

I have been volunteered (i.e, someone decided I volunteered) for a “Debian packaging” tutorial at work, so I decided to give a try at tackling this issue. I also volunteered (voluntarily this time) for a similar talk at RMLL 2011 to make sure I would be forced to do the work and prepare the actual tutorial. I’m also considering teaching this next year in Licence Pro ASRALL, but I haven’t made up my mind about it yet.

The result is a work in progress (hey, I still have a lot of time), but in the release-early-release-often tradition, I’m making it public now in the hope that someone will pick up the idea and do all the work for me (you never know).

I’ve decided to create a set of slides using Latex Beamer. The current version can be found here. The sources are available in a git repository, and all contributions are welcomed (including plain comments or suggestions). The last slide is the current TODO list.

Giving up on Ruby packaging

I have finally reached a decision regarding my involvement in the Debian Ruby packaging efforts. I have decided to stop. This has been a very hard decision to make. I have invested huge amounts of time in that work over the years. I still love the language, and will continue to use it on a daily basis for my own developments. I still hope that it will succeed. I know that some people will be disappointed by that decision (and that others will think “your work was useless anyway, people should use RVM and rubygems”).

But I also know that I won’t be able to push for all the required changes alone. I just don’t have the time, nor the motivation. For the record, here are the changes I would have liked to see in the Ruby community.

The core Ruby development community should mature.
The core Ruby development community is still dominated by Japanese developers. While not a bad thing in itself, it is easily explained by the fact that the main development mailing list, where most of the important decisions are taken, is in japanese. ruby-dev@ should be closed, and all the technical discussions should happen on the english-speaking ruby-core@ list instead.

The release management process should also improve. Currently, it looks like a total mess. The following Ruby development branches are actively maintained:
ruby_1_8 (106 commits over the last six months)
ruby_1_8_6 (4 commits over the last six months)
ruby_1_8_7 (35 commits over the last six months)
ruby_1_9_1 (4 commits over the last six months)
ruby_1_9_2 (227 commits over the last six months)
trunk (1543 commits over the last six months)

While the state of the ruby_1_8_6 and ruby_1_9_1 branches is clear (very important bugfixes only), the state of all of the other branches is rather unclear.
What’s the stable Ruby branch? 1.8 or 1.9? If it’s 1.9, why are people still actively developing in the ruby_1_8 branch? How long will they continue to be maintained in parallel, dividing the manpower? Is a Ruby 1.8.8 release to be expected? Will it be ABI/API compatible with 1.8.7? Is the ruby_1_8_7 branch really bugfixes-only? How much testing of it has been done? If it’s bugfixes-only and regression-free, I should push it to Debian squeeze, due to be released in a few weeks. But would you recommend that? Due to past breakages in the ruby_1_8_7 branch, it’s unlikely that we will do it.
Is the ruby_1_9_2 a regression-free, bugfix-only branch? If yes, isn’t 227 commits over 6 months a lot? What will be the version of the next release of “trunk”? When is it expected? Will it be ABI-compatible with the current ruby_1_9_2 branch? API-compatible?
New releases in the 1.8.7 and 1.9.2 branches were done on december 25th. Why were they no betas or RCs allowing wider testing? How much testing has been done behind the scenes?

Most of those questions have no clear answer. The Ruby development community should build a common understanding of the status of the various branches, and of their release expectations. Releasing on december 25th of each year sounds fun, but is releasing when everybody is on vacation really a good idea?

It would be fantastic to have something similar to Python Enhancement Proposals in the Ruby community. But having open discussions in english about the major issues would already be great.

Ruby is not just the interpreter.
The Ruby development community should clearly define what the Ruby platform is. There are some big players, like Rails, and newer interpreter releases should not be done before ensuring that those big players still work.

Also, since we have alternative Ruby interpreters, like JRuby, Rubinius and MacRuby, we need a clear process on how they integrate with the rest of the ecosystem. For example, having each of them rely on their own outdated fork of the whole stdlib is ridiculous, since it’s not where they compete.

The Ruby community should acknowledge that RVM and Rubygems are not for everybody. People who say so should be laughed at. Of course, RVM and Rubygems are nice tools for some people. But it is completely wrong to believe that compiling from source using RVM should be the standard way of installing Ruby, or that all people interested in installing Redmine should know that Ruby has its own specific packaging system. The Ruby community should work with their target platforms to improve how Ruby is distributed instead of reinventing the wheel. That includes Debian, but also RedHat-based distros, for example. It is likely that it won’t be possible to reach a one-size-fits-all situation. But that’s real life.

Some people in the Ruby community should stop behaving like assholes. As one of the Debian Ruby maintainers, I have been routinely accused of creating crippled packages on purpose (FTR, I don’t think that the Debian packages are crippled, despite what the rumors says). Debian is not the only target of that. Just yesterday, someone called for abandonning YARV (the new Ruby VM in Ruby 1.9), calling it Yet Another Random Vailure. This kind of comments is really hurting the people who are investing their free time in Ruby, and is turning away people who consider getting involved. In Debian, we have had a lot of problems getting people to help with Ruby maintenance since they are getting shit from the community all the time.

So, what’s the future for Ruby in Debian?

  • For the interpreter, the two other maintainers, Akira Yamada and Daigo Moriwaki, are of course free to continue their work, and I wish them good luck.
  • For the pkg-ruby-extras team, which maintains most of the Ruby libraries and applications, the future is less clear. The team was already badly understaffed, and I feel that I should probably clean up its status by orphaning/removing the packages that are unmaintained otherwise. This won’t affect all the packages that the team maintains: some packages (like redmine) are actively maintained, and will stay.
  • For me, it just means I will have more time for other things (possibly not Free Software-related). If things improve dramatically, I might also come back to Ruby packaging at some point.

Update: there’s also a number of interesting comments about this post on this site.
Update 2: First, thanks a lot for all the interesting comments. I will make some follow-up posts trying to summarize what was said. It seems that this post also triggered some reactions on ruby-core@, with Charles Olivier explaining the JRuby stdlib fork, and Yui Naruse clarifying that all questions are welcomed on ruby-core@. This is great, really.

How to demotivate people (Re: Making Debian Responsible For Its Actions)

In my Ruby packaging in Debian and Ubuntu: Mythbusting and FAQ, I wrote The Ruby developers (or expert users) community is generally very hostile towards Debian. Many harsh words and insults are used in discussions mentioning Debian in the ruby-core@ mailing list, and there are frequent recommendations to avoid the Debian packages and install from sources, which is quite demotivating (Actually, what prompted that blog post is someone calling Debian-specific changes unforgivable).

Zed A. Shaw just provided a very good example of that. Quoting the most juicy parts:

It’s simply a tactic to make sure that you are stuck on Debian.

If we look at this problem from an open source perspective, Debian is at a minimum being a horrible citizen.

What is happening is control is not aligned with responsibility. Debian has taken control of the software package and recrafted it for their very narrow set of system administrator users. However, they refuse to take responsibility for the defect and quality problems they create. Instead I end up being responsible for their mistakes, which is wrong.

If Debian is in control, then Debian should be made responsible. That’s the only way to get them to change it. If they don’t want to be responsible, then they should let me be in control.

I’m imagining a mass petition, maybe some funny ad campaigns, in-person confrontations meant to embarrass package maintainers, SEO tricks to get people off Debian, promotion of any alternative to Debian, anything to make them pay, apologize, and listen.

If you write software and you’re sick of Debian screwing up your gear, then shoot me ideas for how we can make this happen. I think that if enough of us band together to make Debian responsible for its actions we can actually get them to start asking us before they package our software to find out how we think it should be packaged.

On the technical part of the post:

The problem that Zed is experiencing is unknown to me, and also to the Debian and Ubuntu bug trackers. To my knowledge, rubygems doesn’t not require openssl or net/scp to work, and a simple grep through the sources kind-of confirms that. But of course, that blog post was written as a rant, not as a proper bug report.

My best guess is that the gem that Zed is installing requires openssl and net/scp. But of course, rubygems can’t just depend on all the libraries that applications or libraries installed via rubygems could possibly require. Regarding openssl, we had a problem in Ruby because of the licensing situation of Ruby (dual licensed GPLv2+ and Ruby license, linked with openssl, which might raise some eyebrows), so it was split off the main standard library package until recently (and, apparently, in the version Zed is using). For net/scp, it is not part of the standard library, and not a dependency of rubygems, so I don’t see why Zed expects it to be installed.

On the rest of the post:

Following my last post, several people volunteered to help with Ruby on Debian (and Ubuntu). In particular, Laurent Arnoud has been doing some fantastic work fixing the pkg-ruby-extras team’s RC bugs. Also, all the messages of support both in the comments of my last post were very much appreciated.

But I’m really wondering why the Ruby community generates so many poisonous people. All my work on Debian is done on a volunteer basis, and it’s really hard to get so much shit from it.

Helping improve Ruby on Debian and Ubuntu

After my last blog post on Ruby in Debian and Ubuntu, I got a few offers for help, which is excellent.

So I’ve decided to update this wiki page with pointers to the various team resources, and some examples of tasks that newcomers can do.

I’d like to stress that many pending tasks are really doable by someone not very qualified in Debian packaging. They are not done just because nobody has the time to do them. Some good examples are the libinotify-ruby, libimage-size-ruby, libffi-ruby RC bugs.