dash as /bin/sh, and now ld –as-needed. Pattern?

I must admit that I’ve never been a big fan of the dash as /bin/sh change. I have three main problems with the switch:

POSIX compliance as an argument

Complying to standards is a really good thing. But when everybody is ignoring the standard because they want the comfort of newer features, maybe it’s a sign that the standard should be updated to include those newer features. Most of the bashims used everywhere in scripts were signifiant improvements, like the ability to write:
cp {short1/path1,short2/path2}/very/long/common/path/to/a/file
instead of:
cp short1/path1/very/long/common/path/to/a/file short2/path2/very/long/common/path/to/a/file

The option to improve bash was not fully explored

We started with the premise that bash is bloated, slow, and cannot be improved. Maybe you can help me with that, but I could only find a few simplistic benchmarks comparing dash and bash, and I could not find any analysis of why bash is slow, and why it cannot be improved.
One of the obvious problems is that bash is also an interactive shell, and is linked to ncurses and terminfo, which increases the startup time. But have we investigated the possibility to have a /bin/bash-non-interactive binary that would not be linked to ncurses?

The change was brought to users

While it is OK for Debian (or Ubuntu, in that case, since that change was done in Ubuntu first) to force its developers to use POSIX-compliant scripts, the switch could have been made only to Debian-created scripts (by switching them from a /bin/sh shebang to a /bin/dash shebang, for example). I have trouble justifying that this change was forced on users as well.

Next: linker changes

… and we are doing it again. A set of linker changes (see also the Ubuntu page) was already done in Ubuntu, and is very likely to be done in Debian as well. This switch requires deep changes in some buildsystems (it requires ordering of libraries and forbids indirect dependencies), and is rather painful (it was reverted before the Ubuntu 11.04 release because it was not possible to fix all the packages during the natty release cycle, but is done in the 11.10 release). Of course, there are justifications for this change. But I’m not sure that it’s worth all the trouble created for users.

Windows dynamic disks and debian installer ?

Windows has a feature called Dynamic Disks, which makes it ignore the DOS partition table and manage partitions its own ways. gparted is supposed to detect that, and warn the user that it can’t deal with that. The solution is to reboot in Windows, switch the partition back to “Basic disks”, and then proceed with installing Debian (or Ubuntu).

However, a student of mine had an interesting experience because gparted did not detect the dynamic disk stuff and did not warn about it, so the student went ahead with installing Debian squeeze. After reboot in Windows, Windows sees the partition as “invalid”. The same happens with Ubuntu 11.04 (not really surprising).

I’m surprised I did not find much information about that issue. Does someone has more info ?

The student was using two disks in his laptop, and only the second one (not the one used to boot windows) was using “dynamic disks”.

Debconf

So, the fifth Debconf I’ve had the chance to attend is clearly over now. It was great, and all organizers (and sponsors) really deserve huge “thank you” for making this event so successful. I’m already looking forward to next year’s edition.

Debconf has been very productive for me. I chaired 4 sessions:

  • the usual Quality Assurance BOF, which was a bit depressing: even if work gets done, the QA “team” doesn’t really feel like a “team”. Maybe that’s because a QA “team” is not needed, and we should instead split it into smaller teams focusing on subsets of the QA work (archive-wide testing, QA services, etc.)?
  • a BOF on Ruby, where I demoed the work we have been doing around gem2deb, our new dh-based packaging helper [Ruby team website]
  • a tutorial on archive testing, hoping to get more people involved in tests such as archive rebuilds. If you are interested in helping with reporting bugs, please drop me an email [doc about archive tests on wiki.d.o]
  • a BOF on finding a ‘standard’ Git workflows for packaging teams. More work is needed on this, but it looks like a good start. [thread on -devel@, wiki page, BOF notes]

I’ve also made numerous uploads of Ruby-related packages, and reduced my backlog on UDD to a reasonable level. I even managed to make a developers-reference upload, integrating all the pending patches from the BTS.

In other news, I’m very excited about the recent progress on expo.debian.net (a mentors.debian.net replacement), which could help streamline our sponsorship process.

Going to RMLL (LSM) and Debconf!

Next week, I’ll head to Strasbourg for Rencontres Mondiales du Logiciel Libre 2011. On monday morning, I’ll be giving my Debian Packaging Tutorial for the second time. Let’s hope it goes well and I can recruit some future DDs!

Then, at the end of July, I’ll attend Debconf again. Unfortunately, I won’t be able to participate in Debcamp this year, but I look forward to a full week of talks and exciting discussions. There, I’ll be chairing two sessions about Ruby in Debian and Quality Assurance.

Packaging tutorial update

This week I finally used my packaging tutorial slides myself, a few days after Phil Hands used them at a UKUUG event.

The tutorial is now available as a Debian package (in Debian unstable) named packaging-tutorial, and I’ve talked to debian-doc@ about linking it from appropriate places. It’s also hosted on collab-maint now, so don’t hesitate to contribute.

Links :

Changes to Ruby in Debian (and Ubuntu)

(this is a copy of an mail sent to ruby-core@ and ruby-talk@)

Since the beginning of 2011, the Debian Ruby team has been working on several big changes. Those changes all are available in Debian unstable, some of them are also available in Debian testing, and they should all be available in the next Debian and Ubuntu releases.

I think that it addresses most of the reasonable concerns about Ruby in Debian.

Using alternatives to switch between Ruby implementations

The alternatives system is now used to manage the “ruby” symlink and the other related symlinks, making it easy to switch between Ruby implementations (only Ruby 1.8 and 1.9.X at the moment) (see this mail for details). The default choice for Ruby is still 1.8, but this change will make it easy for us to make a switch to 1.9.X by default (likely by the release of Debian wheezy).

Installing gems executables to /usr/local/bin

Rubygems (both as a standalone package, and as shipped with Ruby 1.9.X) now install executables to /usr/local/bin. (The other files still get installed under /var, see this mail)

Enabling gem update –system

gem update –system has been re-enabled. Since upgrading rubygems to a version that may not have been properly tested with the rest of the Debian system may cause issues in the user’s system, there’s a big warning about that. The user can confirm and upgrade rubygems anyway by defining an environment variable.

New gem2deb packaging helper

There’s a new packaging helper, named gem2deb, that makes it very easy to generate Debian source packages from Rubygems. We are in the process of migrating all ruby libraries packaged in Debian to that new helper. It will take some time, though (help is welcomed).

Transition status: http://pkg-ruby-extras.alioth.debian.org/wheezy/

One big benefit of the switch to gem2deb for the Ruby community is that, in the process, we are enabling test suites at build time for each package and each Ruby implementation. This should make it easy to detect regressions in new interpreter versions.

Ruby 1.9.3

We will switch to Ruby 1.9.3 ASAP (probably when it is branched off trunk, with a package first in Debian experimental). Since the Ruby compatibility version issue is likely to stay around, we will re-evaluate how we are dealing with it (to avoid the ruby1.9.1 package <=> ruby -v = 1.9.2 problem that confuses many users). This is likely by switching the package name to ruby1.9.3 (keep a ruby1.9.1 package for compatibility). The package containing the shared library will stay libruby1.9.1.

Links

Debian Packaging Tutorial update

As previously announced, I’ve been working on a Debian packaging tutorial. It is composed of about 60 slides providing a throughout overview of Debian packaging.

It now reached the point where I consider it ready for use, and I am looking forward to reviews and comments.
The document is split into 4 different PDFs:

And of course, it can be found on git.debian.org:
git clone git://git.debian.org/collab-maint/packaging-tutorial.git

Debian rolling discussion on -devel@

Here is my attempt at a summary of the rolling discussion currently happening on debian-devel@. It might not be complete, it’s probably a bit biased, but I hope that it’s still better than nothing. It was also posted on debian-devel@.

If you are involved in Debian development, please discuss it on debian-devel@, rather than in the comments of this blog.

If you want to provide feedback about the Debian rolling idea, use this Doodle poll and/or the comments of this blog.
Update : Someone had the good idea to vandalize the poll, and remove all the votes from people in favor of rolling. Before the vadalism (log), the results were:
37 votes: I am a Debian user, I use testing or unstable, and I would like rolling to happen.
6 votes: I am a Debian user, I use testing or unstable, and I DON’T want rolling to happen.
17 votes:  I am a Debian user, I use stable, and I would like rolling to happen.
3 votes: I am a Debian user, I use stable, and I DON’T want rolling to happen.
7 votes: I am NOT a Debian user, but I would probably use it if rolling happened.
1 vote: I am NOT a Debian user, and I don’t care about rolling.

Motivations

There’s some user demand for rolling releases. For evidence, one can look at the usage of Debian testing or unstable which clearly goes further than the Debian development community. Or at the quickly growing market share of ArchLinux. Or at the interest in LinuxMint and Aptosid. Or at the DPL’s report of his interactions with the press.

Debian’s only official product is its stable releases. While it’s a clearly great and useful product, many users and developers don’t recognize themselves in it: it contains software that is too old for their needs. The success of Ubuntu is related to this problem: Ubuntu proposes a different compromise between stability and up-to-date-ness.

While concurrencing Ubuntu with more frequent stable releases (released every 6-months, for example) doesn’t look like the right thing to do, Debian is in a perfect position to provide a rolling release with marginal additional work, since Debian already has testing (and unstable) to build on.

The rolling discussion investigates how we could endorse the concept of a rolling release, provided as a product in addition to stable releases. This rolling release would be based on the current testing branch.

Benefits for Debian:

  • Attract users who think that testing is only a development branch, and want newer software than what one finds in stable. Those users are likely to be rather advanced users (free software developers and contributors), thus interesting to work with (able to submit good-quality bug reports, etc). Some of them could also become Debian contributors. And even if they don’t, more users of testing/rolling means more testers of the next stable release [remember how the bug reporting rate of Ubuntu is higher than Debian’s — some areas of Debian could use more testers].
  • Give back to the free software world by providing a platform where new upstream releases would quickly be available to users. Since users would be able to test new upstream releases earlier, they would be able to provide feedback to upstream devs earlier, contributing to a shorter feedback loop. Debian is often identified by upstream developers as the platform with releases from two years ago. I would love to see Debian in a position to contribute more to improviing the quality of the Free Software world.
  • Get back some attention that is currently taken away from Debian by derivatives. This is important to carry our political or technical messages, which are not necessarily carried out by our derivatives.

Current proposed plan for rolling

(Disclaimer: this is mostly my view. It is shared by others, but some details might not be)

rolling is mostly about (external) communication. It is not expected to change our development processes fundamentally.
It would be a statement by the project (through a GR, for example). A very preliminary draft was proposed by Raphael Hertzog:

Title: Debian endorses usage of testing by end-users, and renames it to rolling

The Debian project recognizes that the Debian testing distribution—initially created to make it easier to prepare and test the next stable release—has become a useful product of its own. It satisfies the needs of users who are looking for the latest stable versions of software and who can cope (or even appreciate) a system that’s constantly evolving.

The Debian project decides to endorse this usage and will strive to provide a good experience to users of “testing”. To better communicate this policy change to our users, “testing” will be renamed “rolling”.

Yes, it’s mostly “PR bullshit”, and I don’t expect it to significantly change Debian development processes. However, communication is necessary if we want to attract new users. What would change is more attention from developers to what happens in testing/rolling, which is a good thing since a better testing/rolling makes it easier to create stable releases from it.

Open questions

Most of the discussion is about the influence of the introduction of ‘rolling’ on Debian development processes, and in particular, on the painful process resulting in stable releases. Many fear that, with ‘rolling’, it will be harder to get stable releases out.

The root question is: if we do rolling, what do we do during freezes? Several options have been mentioned in the thread:

  1. We could decide not to do anything special: just freeze rolling for ~6 months, as we used to freeze testing. That might bore people who like the constant flux of new upstream releases, but if we decide that it’s the only way to ensure high-quality stable releases, so be it.
  2. We could decide to fork a frozen branch when we freeze, and continue to manage rolling using migrations from unstable.
  3. We could mix both solutions: freeze rolling for 3 months, so that most of the stabilization work occurs with a single active branch, and then, for the final release preparation, fork frozen off rolling, and unfreeze rolling.

Two kinds of objections have been raised:

  • Those against the rolling concept:
    • “It’s only about PR, Debian isn’t about PR.” Answer: PR does matter sometimes, especially if we want to attract users.
    • “There’s usually no installer for it, other than installing the latest stable release and dist-upgrading, which doesn’t always work.” Answer: True; but it sounds like an acceptable problem. And if upgrades from the latest stable fail, it’s an RC bug, so we would like to know. And if we do get d-i betas, it’s a great way to get user testing for them.
    • It will split the developers base between supporting ‘rolling’ and supporting stable releases (which also need to be supported after they have been released). Answer: already the case today.
    • Testing is not targeted at end users, but is a tool for the release team to create stable releases. It needs to stay that way. Answer: really, can’t we do both?
    • Renaming testing to rolling will require changes in many parts of Debian infrastructure. Answer: some problems can be mitigated by keeping a testing symlink. The remaining impact needs to be evaluated.
  • Those against the two development branches during freezes problem:
    • It splits the users and developers base between two branches (less users means less bug reports ; less developers means less bug fixing). Answer: true.
    • It requires the use of two different entry points for packages (unstable for packages targeted at rolling, frozen-proposed-updates for packages targeted at frozen). Answer: true.
    • All in all, huge overhead for the release team. Answer: true.
    • Overhead for developers, who need to support two targets. Answer: true.
    • Just after a stable release, we would start the next release from rolling, instead of stable. So we would start from a “less clean” base. Answer: true.
    • It’s even worse if you consider staged freezes (freezing base packages earlier than the rest of the distribution, for example). Answer: true; but is it really a problem if some base packages are frozen in rolling for a few months?

Other things that were discussed:

  • possible changes to processes around testing to make it more usable (reduce the set of architectures required for migration to testing ; allow/encourage usage of t-p-u to rebuild unstable packages that are ready to transition except for the fact that they are entangled in a transition ; have different level of RC bugs: there are RC bugs that are acceptable in rolling that are not acceptable in stable)
  • PPAs for Debian
  • Developer activity during freeze (developer temporarily stopping to work on Debian during freezes)
  • Let’s improve our packaging process or reduce the duration of our freezes before introducing rolling. Answer: but then, shouldn’t we also stop doing stable releases for a while? ;)
  • Setting up an official “rolling instance”. Answer: that can’t work. detailed answer
  • Using unstable instead of testing as the basis for rolling. Answer: there seems to be more demand for something similar to testing, than for something similar to unstable.

Tentative personal conclusion

I have the impression that advertising testing as a rolling release usable by end-users is generally considered a good thing.
The renaming of testing to rolling is not as consensual, but most opponents have a “whatever ; if you want” position.

How we deal with freezes is the hard point in this discussion. I’m personnally in favor of the “freeze rolling for 3 months, then fork frozen and unfreeze rolling” plan, though it has some problems too (it is not clear whether the required manpower really decreases at the end of freezes).

Where do we go from there? After another round of discussion, we might postpone the “how to deal with freezes” question to later, and proceed with a GR to measure the support for the “testing for end-users + s/testing/rolling/” proposal.

Banshee and Ubuntu

[ If you haven’t heard of this debate yet, you probably want to read Vincent Untz’s and Mark Shuttleworth’s blog posts. ]

Last year, I gave a talk at FOSDEM about Debian and Ubuntu (slides of a slightly updated version). One of my points was that Debian has better values, being a volunteer-driven project where decisions are taken in the open.

In contrast, Ubuntu is a project managed and controlled by Canonical, and recent history has shown that Canonical had no problem imposing some decisions to the developers community: first with the inclusion of UbuntuOne, then the switch from Google to Yahoo! to Google as the default search engine, both to increase revenue streams.

So one should not be surprised by the Banshee story. I find Mark’s justification quite difficult to buy, and similar to Apple’s model where 30% of the revenues from the App Store go to Apple, and 70% to the seller of the application.

For those wondering how much work was done by Canonical directly on the Banshee package: the banshee package in Ubuntu natty is based on the package currently in Debian experimental. The package is mainly maintained in Debian by an Ubuntu developer not paid by Canonical AFAIK, Chow Loong Jin. There are some differences between the Debian package and the Ubuntu package, which are fairly limited (full diff here) and mainly about enabling UbuntuOne and disabling the other music stores. That patch itself apparently was provided by Jo Shields, who doesn’t seem to be a canonical employee. (Feel free to correct me)

I think that one of the conclusions to draw from this story is that we now have several proofs that Ubuntu isn’t a volunteer-driven project, and that volunteer contributors should really decide whether they are OK with working for free for Canonical, or if their free time would be better spent on other projects where they actually have a chance to influence decisions. From the Debian POV, I’m still convinced that we should take the feedback that we receive from Ubuntu in consideration to improve our Debian packages (by looking at patches made by Ubuntu, or at bugs reported in Launchpad). But my motivation for contributing to Ubuntu directly has just diminished a bit more (not that it was very high before).