May 2nd, 2011 by lucas
Here is my attempt at a summary of the rolling discussion currently happening on debian-devel@. It might not be complete, it’s probably a bit biased, but I hope that it’s still better than nothing. It was also posted on debian-devel@.
If you are involved in Debian development, please discuss it on debian-devel@, rather than in the comments of this blog.
If you want to provide feedback about the Debian rolling idea, use this Doodle poll and/or the comments of this blog.
Update : Someone had the good idea to vandalize the poll, and remove all the votes from people in favor of rolling. Before the vadalism (log), the results were:
37 votes: I am a Debian user, I use testing or unstable, and I would like rolling to happen.
6 votes: I am a Debian user, I use testing or unstable, and I DON’T want rolling to happen.
17 votes: I am a Debian user, I use stable, and I would like rolling to happen.
3 votes: I am a Debian user, I use stable, and I DON’T want rolling to happen.
7 votes: I am NOT a Debian user, but I would probably use it if rolling happened.
1 vote: I am NOT a Debian user, and I don’t care about rolling.
There’s some user demand for rolling releases. For evidence, one can look at the usage of Debian testing or unstable which clearly goes further than the Debian development community. Or at the quickly growing market share of ArchLinux. Or at the interest in LinuxMint and Aptosid. Or at the DPL’s report of his interactions with the press.
Debian’s only official product is its stable releases. While it’s a clearly great and useful product, many users and developers don’t recognize themselves in it: it contains software that is too old for their needs. The success of Ubuntu is related to this problem: Ubuntu proposes a different compromise between stability and up-to-date-ness.
While concurrencing Ubuntu with more frequent stable releases (released every 6-months, for example) doesn’t look like the right thing to do, Debian is in a perfect position to provide a rolling release with marginal additional work, since Debian already has testing (and unstable) to build on.
The rolling discussion investigates how we could endorse the concept of a rolling release, provided as a product in addition to stable releases. This rolling release would be based on the current testing branch.
Benefits for Debian:
- Attract users who think that testing is only a development branch, and want newer software than what one finds in stable. Those users are likely to be rather advanced users (free software developers and contributors), thus interesting to work with (able to submit good-quality bug reports, etc). Some of them could also become Debian contributors. And even if they don’t, more users of testing/rolling means more testers of the next stable release [remember how the bug reporting rate of Ubuntu is higher than Debian's -- some areas of Debian could use more testers].
- Give back to the free software world by providing a platform where new upstream releases would quickly be available to users. Since users would be able to test new upstream releases earlier, they would be able to provide feedback to upstream devs earlier, contributing to a shorter feedback loop. Debian is often identified by upstream developers as the platform with releases from two years ago. I would love to see Debian in a position to contribute more to improviing the quality of the Free Software world.
- Get back some attention that is currently taken away from Debian by derivatives. This is important to carry our political or technical messages, which are not necessarily carried out by our derivatives.
Current proposed plan for rolling
(Disclaimer: this is mostly my view. It is shared by others, but some details might not be)
rolling is mostly about (external) communication. It is not expected to change our development processes fundamentally.
It would be a statement by the project (through a GR, for example). A very preliminary draft was proposed by Raphael Hertzog:
Title: Debian endorses usage of testing by end-users, and renames it to rolling
The Debian project recognizes that the Debian testing distribution—initially created to make it easier to prepare and test the next stable release—has become a useful product of its own. It satisfies the needs of users who are looking for the latest stable versions of software and who can cope (or even appreciate) a system that’s constantly evolving.
The Debian project decides to endorse this usage and will strive to provide a good experience to users of “testing”. To better communicate this policy change to our users, “testing” will be renamed “rolling”.
Yes, it’s mostly “PR bullshit”, and I don’t expect it to significantly change Debian development processes. However, communication is necessary if we want to attract new users. What would change is more attention from developers to what happens in testing/rolling, which is a good thing since a better testing/rolling makes it easier to create stable releases from it.
Most of the discussion is about the influence of the introduction of ‘rolling’ on Debian development processes, and in particular, on the painful process resulting in stable releases. Many fear that, with ‘rolling’, it will be harder to get stable releases out.
The root question is: if we do rolling, what do we do during freezes? Several options have been mentioned in the thread:
- We could decide not to do anything special: just freeze rolling for ~6 months, as we used to freeze testing. That might bore people who like the constant flux of new upstream releases, but if we decide that it’s the only way to ensure high-quality stable releases, so be it.
- We could decide to fork a frozen branch when we freeze, and continue to manage rolling using migrations from unstable.
- We could mix both solutions: freeze rolling for 3 months, so that most of the stabilization work occurs with a single active branch, and then, for the final release preparation, fork frozen off rolling, and unfreeze rolling.
Two kinds of objections have been raised:
- Those against the rolling concept:
- “It’s only about PR, Debian isn’t about PR.” Answer: PR does matter sometimes, especially if we want to attract users.
- “There’s usually no installer for it, other than installing the latest stable release and dist-upgrading, which doesn’t always work.” Answer: True; but it sounds like an acceptable problem. And if upgrades from the latest stable fail, it’s an RC bug, so we would like to know. And if we do get d-i betas, it’s a great way to get user testing for them.
- It will split the developers base between supporting ‘rolling’ and supporting stable releases (which also need to be supported after they have been released). Answer: already the case today.
- Testing is not targeted at end users, but is a tool for the release team to create stable releases. It needs to stay that way. Answer: really, can’t we do both?
- Renaming testing to rolling will require changes in many parts of Debian infrastructure. Answer: some problems can be mitigated by keeping a testing symlink. The remaining impact needs to be evaluated.
- Those against the two development branches during freezes problem:
- It splits the users and developers base between two branches (less users means less bug reports ; less developers means less bug fixing). Answer: true.
- It requires the use of two different entry points for packages (unstable for packages targeted at rolling, frozen-proposed-updates for packages targeted at frozen). Answer: true.
- All in all, huge overhead for the release team. Answer: true.
- Overhead for developers, who need to support two targets. Answer: true.
- Just after a stable release, we would start the next release from rolling, instead of stable. So we would start from a “less clean” base. Answer: true.
- It’s even worse if you consider staged freezes (freezing base packages earlier than the rest of the distribution, for example). Answer: true; but is it really a problem if some base packages are frozen in rolling for a few months?
Other things that were discussed:
- possible changes to processes around testing to make it more usable (reduce the set of architectures required for migration to testing ; allow/encourage usage of t-p-u to rebuild unstable packages that are ready to transition except for the fact that they are entangled in a transition ; have different level of RC bugs: there are RC bugs that are acceptable in rolling that are not acceptable in stable)
- PPAs for Debian
- Developer activity during freeze (developer temporarily stopping to work on Debian during freezes)
- Let’s improve our packaging process or reduce the duration of our freezes before introducing rolling. Answer: but then, shouldn’t we also stop doing stable releases for a while? ;)
- Setting up an official “rolling instance”. Answer: that can’t work. detailed answer
- Using unstable instead of testing as the basis for rolling. Answer: there seems to be more demand for something similar to testing, than for something similar to unstable.
Tentative personal conclusion
I have the impression that advertising testing as a rolling release usable by end-users is generally considered a good thing.
The renaming of testing to rolling is not as consensual, but most opponents have a “whatever ; if you want” position.
How we deal with freezes is the hard point in this discussion. I’m personnally in favor of the “freeze rolling for 3 months, then fork frozen and unfreeze rolling” plan, though it has some problems too (it is not clear whether the required manpower really decreases at the end of freezes).
Where do we go from there? After another round of discussion, we might postpone the “how to deal with freezes” question to later, and proceed with a GR to measure the support for the “testing for end-users + s/testing/rolling/” proposal.
March 2nd, 2011 by lucas
Last year, I gave a talk at FOSDEM about Debian and Ubuntu (slides of a slightly updated version). One of my points was that Debian has better values, being a volunteer-driven project where decisions are taken in the open.
In contrast, Ubuntu is a project managed and controlled by Canonical, and recent history has shown that Canonical had no problem imposing some decisions to the developers community: first with the inclusion of UbuntuOne, then the switch from Google to Yahoo! to Google as the default search engine, both to increase revenue streams.
So one should not be surprised by the Banshee story. I find Mark’s justification quite difficult to buy, and similar to Apple’s model where 30% of the revenues from the App Store go to Apple, and 70% to the seller of the application.
For those wondering how much work was done by Canonical directly on the Banshee package: the banshee package in Ubuntu natty is based on the package currently in Debian experimental. The package is mainly maintained in Debian by an Ubuntu developer not paid by Canonical AFAIK, Chow Loong Jin. There are some differences between the Debian package and the Ubuntu package, which are fairly limited (full diff here) and mainly about enabling UbuntuOne and disabling the other music stores. That patch itself apparently was provided by Jo Shields, who doesn’t seem to be a canonical employee. (Feel free to correct me)
I think that one of the conclusions to draw from this story is that we now have several proofs that Ubuntu isn’t a volunteer-driven project, and that volunteer contributors should really decide whether they are OK with working for free for Canonical, or if their free time would be better spent on other projects where they actually have a chance to influence decisions. From the Debian POV, I’m still convinced that we should take the feedback that we receive from Ubuntu in consideration to improve our Debian packages (by looking at patches made by Ubuntu, or at bugs reported in Launchpad). But my motivation for contributing to Ubuntu directly has just diminished a bit more (not that it was very high before).
February 25th, 2011 by lucas
I’ve been playing with snapshot.d.o, which is a fantastic resource to look at how Debian has been changing over the years.
Below are some preliminary results about various aspects of source packages: format, packaging helpers, and patch systems.
- “modif-files” indicates packages that do not use a patch system, but modify files outside debian/.
- “no-modif-files” indicates packages that do not use a patch system, but do not modify files outside debian/.
- “other” is “everything else”. Like packages using dpatch, but still modifying some files outside Debian. Or using both dpatch and quilt. (yes, some packages do that).
- People have been moving away from simple-patchsys and dpatch to quilt since 01/2009 (their usage has been decreasing since then). It seems that discussing the introduction of the new package formats was enough to make people move away from dpatch, even if the 3.0 (quilt) was not usable yet. We now have less than 1000 packages still using dpatch or simple-patchsys. Can we get rid of them all for wheezy? It would be nice to standardize on quilt.
- The growth of the 3.0 (quilt) format is impressive. Its adoption is much faster than the adoption of dh7. That surprises me a lot, since I had the impression that more people were unhappy about 3.0 (quilt) than about dh7.
- The number of packages with no packaging helper, but no modified files outside debian/ is dropping. Either than means that we are patching upstream more and more (unlikely), or that means that people are switching to 3.0 (quilt) even if they have no patches, probably due to the other benefits! Could this mean that we could standardize on 3.0 (quilt) and 3.0 (native) for wheezy or wheezy+1?
- The number of packages that modify files outside debian/ without using a patch system has been constantly dropping since 2005.
- dh7 is not a cdbs killer. While it seems to be the cause for a huge drop in “classic debhelper” packaging, the number of packages using cdbs is still progressing.
If you have other ideas of things to investigate using snapshot.d.o, don’t hesitate to ask in comments.
February 21st, 2011 by lucas
We have many orphaned packages in Debian. Those are packages that no longer have a maintainer willing to care for them. They are a problem for several reasons. Often, there have better alternatives (that’s often why the maintainer gave up maintenance), but users keep installing them because they show up in apt-cache search. Or, they might be broken, without anybody noticing because they only have a few users. Also, developers need to take them into account. They might slow down transitions, have RC bugs that need fixing, etc.
So, there are good reasons to remove some of the orphaned packages from the Debian archive. Not all of them. But those that have been orphaned for a long time without anybody stepping up to take over the maintenance, have very few users according to popcon, etc. Removing some of them also makes it easier to expose the ones that should really find an adopter.
Also, that removal is not final. The good reason to do this now is that we are at the beginning of a release cycle, and that it will be easy to reintroduce packages that are really needed during the next two years. Also, packages (both source and binaries) are still available on snapshot.debian.org, so users can still install them from there, and re-uploading them to Debian is very easy if someone wants to adopt them.
So, how do I proceed? I use bapase, and go through specific sets of packages (using a custom version of bapase to restrict the list to those packages). For example, orphaned packages that have a popcon lower than 50, have been orphaned for more than 500 days, and whose orphaning bugs have not been modified in the last 100 days. Then, I read the bug log, and decide if there’s a reason to keep the package. If there’s not, I request the removal.
As of today, there are 471 orphaned packages in Debian. The goal is not to get down to zero, but it’s certainly possible to get down to 200 or 250.
How can you help? Start from the list, and find some easy targets. You can take a look at bug #483252 if you want to re-use my template. Also, one thing that is currently missing in the process is that orphaned packages that have reverse-dependencies should not be removed without removing their dependencies (if appropriate). An easy and efficient way to know if a given package has reverse (build-)?dependencies would be great, so I could integrate this info into bapase and exclude those packages from the list, or mark them somehow.
February 17th, 2011 by lucas
One of the common complains about Debian packaging is that it’s hard to learn because, while there is quite a lot of high-quality documentation, it is often written more as a reference manual than as a tutorial: it’s great if you already know everything and want to check some detail, but not so great if you want to learn everything from scratch.
I have been volunteered (i.e, someone decided I volunteered) for a “Debian packaging” tutorial at work, so I decided to give a try at tackling this issue. I also volunteered (voluntarily this time) for a similar talk at RMLL 2011 to make sure I would be forced to do the work and prepare the actual tutorial. I’m also considering teaching this next year in Licence Pro ASRALL, but I haven’t made up my mind about it yet.
The result is a work in progress (hey, I still have a lot of time), but in the release-early-release-often tradition, I’m making it public now in the hope that someone will pick up the idea and do all the work for me (you never know).
I’ve decided to create a set of slides using Latex Beamer. The current version can be found here. The sources are available in a git repository, and all contributions are welcomed (including plain comments or suggestions). The last slide is the current TODO list.
February 15th, 2011 by lucas
It seems that Matthew Palmer misread my blog post as a complaint against developers asking for patches in exchange of pet feature requests. He really should pay more attention, since I gave “pet feature requests” as an example of case where it would be appropriate to ask for a patch:
Of course, there are cases where it’s perfectly reasonable to ask for a patch: when the task is expected to take hours, or when the result is of limited interest to everybody except the demander.
But even then, it’s not clear. This morning I got an email from a someone involved in PHP packages maintenance, who said that Bugs Search @ UDD was a great tool, but that he would love to have a way to list all bugs affecting packages with the implemented-in::php debtag.
To produce a working patch for this would probably take him at least an hour. You need to set up a copy of the CGI on alioth, understand the DB structure, dig into the code, etc. If you don’t understand SQL and Ruby, it could be a really difficult process. Also, it’s probably quite uninteresting for him to do that, since he is unlikely to stick around developing UDD.
Instead, it didn’t take me more than 5 minutes to produce a one-liner.
The net result for Debian in that case? 55 minutes saved by a developer.
Torsten Werner wrote an angry reply to my post. It’s true that yesterday’s episode triggered my blog post, because I felt quite frustrated to have to provide a patch for something that simple, and would have preferred to use the time for a Debian task where I would be more efficient. But I was not particularly angry at that episode, since that’s something I’ve seen on several occasions. That’s also why I did not mention any team in particular.
The feature request I was making was reasonable, and cannot really be considered a pet feature request (though I might be biased with my QA hat on): mentionning in the dak templates used for bug closure that packages removed from Debian can still be found on snapshot.d.o. The fact that he thinks that addressing this himself turns him into a slave raises interesting questions.
February 15th, 2011 by lucas
There’s a frequent pattern in the Debian community where someone would suggest an improvement to some package or service, and the person responsible for it would reply with “please send a patch”.
Of course, there are cases where it’s perfectly reasonable to ask for a patch: when the task is expected to take hours, or when the result is of limited interest to everybody except the demander.
But often, it would be much more efficient for the person responsible for the service to take 5 or 10 minutes to make the change, than for the demander to spend half an hour learning how to contribute to this service, writing an untestable patch, and sending it back to the person responsible for the service for integration.
It is important to see this problem as the issue of maximizing the usefulness of the resources we have. Often, we are in a situation where we could either:
- spend 30 minutes on a service we don’t know, just to push one change
- spend 30 minutes making three or more changes that others would like to see in a service we know
It would be much better if, instead of saying “Please send a patch”, people would say “Are you interested in contributing a patch, or should I make the change myself?”. Remember that each time you ask someone to take some time to contribute a patch in an area where he is not an efficient contributor, you take some time away from him that could be used to contribute something in an area where he is efficient.
February 6th, 2011 by lucas
Following Our Leader, I’ve just switched to using notmuch with mutt for searching (instead of maildir-utils, previously). I strongly recommend this setup to all mutt users.
As a bonus, accented characters (é, è, …) are now handled correctly while searching, which is quite useful in french.
January 2nd, 2011 by lucas
I have finally reached a decision regarding my involvement in the Debian Ruby packaging efforts. I have decided to stop. This has been a very hard decision to make. I have invested huge amounts of time in that work over the years. I still love the language, and will continue to use it on a daily basis for my own developments. I still hope that it will succeed. I know that some people will be disappointed by that decision (and that others will think “your work was useless anyway, people should use RVM and rubygems”).
But I also know that I won’t be able to push for all the required changes alone. I just don’t have the time, nor the motivation. For the record, here are the changes I would have liked to see in the Ruby community.
The core Ruby development community should mature.
The core Ruby development community is still dominated by Japanese developers. While not a bad thing in itself, it is easily explained by the fact that the main development mailing list, where most of the important decisions are taken, is in japanese. ruby-dev@ should be closed, and all the technical discussions should happen on the english-speaking ruby-core@ list instead.
The release management process should also improve. Currently, it looks like a total mess. The following Ruby development branches are actively maintained:
ruby_1_8 (106 commits over the last six months)
ruby_1_8_6 (4 commits over the last six months)
ruby_1_8_7 (35 commits over the last six months)
ruby_1_9_1 (4 commits over the last six months)
ruby_1_9_2 (227 commits over the last six months)
trunk (1543 commits over the last six months)
While the state of the ruby_1_8_6 and ruby_1_9_1 branches is clear (very important bugfixes only), the state of all of the other branches is rather unclear.
What’s the stable Ruby branch? 1.8 or 1.9? If it’s 1.9, why are people still actively developing in the ruby_1_8 branch? How long will they continue to be maintained in parallel, dividing the manpower? Is a Ruby 1.8.8 release to be expected? Will it be ABI/API compatible with 1.8.7? Is the ruby_1_8_7 branch really bugfixes-only? How much testing of it has been done? If it’s bugfixes-only and regression-free, I should push it to Debian squeeze, due to be released in a few weeks. But would you recommend that? Due to past breakages in the ruby_1_8_7 branch, it’s unlikely that we will do it.
Is the ruby_1_9_2 a regression-free, bugfix-only branch? If yes, isn’t 227 commits over 6 months a lot? What will be the version of the next release of “trunk”? When is it expected? Will it be ABI-compatible with the current ruby_1_9_2 branch? API-compatible?
New releases in the 1.8.7 and 1.9.2 branches were done on december 25th. Why were they no betas or RCs allowing wider testing? How much testing has been done behind the scenes?
Most of those questions have no clear answer. The Ruby development community should build a common understanding of the status of the various branches, and of their release expectations. Releasing on december 25th of each year sounds fun, but is releasing when everybody is on vacation really a good idea?
It would be fantastic to have something similar to Python Enhancement Proposals in the Ruby community. But having open discussions in english about the major issues would already be great.
Ruby is not just the interpreter.
The Ruby development community should clearly define what the Ruby platform is. There are some big players, like Rails, and newer interpreter releases should not be done before ensuring that those big players still work.
Also, since we have alternative Ruby interpreters, like JRuby, Rubinius and MacRuby, we need a clear process on how they integrate with the rest of the ecosystem. For example, having each of them rely on their own outdated fork of the whole stdlib is ridiculous, since it’s not where they compete.
The Ruby community should acknowledge that RVM and Rubygems are not for everybody. People who say so should be laughed at. Of course, RVM and Rubygems are nice tools for some people. But it is completely wrong to believe that compiling from source using RVM should be the standard way of installing Ruby, or that all people interested in installing Redmine should know that Ruby has its own specific packaging system. The Ruby community should work with their target platforms to improve how Ruby is distributed instead of reinventing the wheel. That includes Debian, but also RedHat-based distros, for example. It is likely that it won’t be possible to reach a one-size-fits-all situation. But that’s real life.
Some people in the Ruby community should stop behaving like assholes. As one of the Debian Ruby maintainers, I have been routinely accused of creating crippled packages on purpose (FTR, I don’t think that the Debian packages are crippled, despite what the rumors says). Debian is not the only target of that. Just yesterday, someone called for abandonning YARV (the new Ruby VM in Ruby 1.9), calling it Yet Another Random Vailure. This kind of comments is really hurting the people who are investing their free time in Ruby, and is turning away people who consider getting involved. In Debian, we have had a lot of problems getting people to help with Ruby maintenance since they are getting shit from the community all the time.
So, what’s the future for Ruby in Debian?
- For the interpreter, the two other maintainers, Akira Yamada and Daigo Moriwaki, are of course free to continue their work, and I wish them good luck.
- For the pkg-ruby-extras team, which maintains most of the Ruby libraries and applications, the future is less clear. The team was already badly understaffed, and I feel that I should probably clean up its status by orphaning/removing the packages that are unmaintained otherwise. This won’t affect all the packages that the team maintains: some packages (like redmine) are actively maintained, and will stay.
- For me, it just means I will have more time for other things (possibly not Free Software-related). If things improve dramatically, I might also come back to Ruby packaging at some point.
Update: there’s also a number of interesting comments about this post on this site.
Update 2: First, thanks a lot for all the interesting comments. I will make some follow-up posts trying to summarize what was said. It seems that this post also triggered some reactions on ruby-core@, with Charles Olivier explaining the JRuby stdlib fork, and Yui Naruse clarifying that all questions are welcomed on ruby-core@. This is great, really.
December 25th, 2010 by lucas
(In short: bug fixed upstream, but affecting the version in Debian. Using the package from Ubuntu works.)