May 30th, 2012 by lucas
I like to think that archive rebuilds play an important role in Debian Quality Assurance and Release Management efforts. By trying to rebuild every Debian package from source, one can identify packages that do not build anymore due to changes in other packages (compilers, interpreters, libraries, …). It is also a good way to stress-test all packages that are involved in building other packages.
Since 2007, I had been running Debian archive rebuilds on the Grid’5000 testbed, a research infrastructure for performing experiments on distributed systems – HPC/Grid/Cloud/P2P. I filed more than 6000 release-critical bugs in the process.
Late last year, Amazon kindly offered us a grant to allow us to run such QA tests on Amazon Web Services. With Sébastien Badia, we ported the rebuild infrastructure to AWS (scripts), and several rebuilds have already been carried out on AWS.
On the technical level, 50 to 100 EC2 spot instances are started, and then controlled from a master instance using SSH. On build instances, a classic sbuild setup is used. Logs are retrieved to the master node after rebuilds, and build instances are simply shut down when there are no more tasks to process. Several tasks are processed simultaneously on each instance, and when they fail, they are retried again with no other concurrent build on the same instance, to eliminate random failures caused by load or timing issues. All the scripts are designed to support other kind of QA tests, not just rebuilds.
Moving to Amazon Web Services will facilitate sharing the human workload of doing those tests. It is now possible for developers interested in custom tests to do them themselves (hint hint).
April 13th, 2012 by lucas
I’ve just updated the Debian Packaging Tutorial. This new version addresses a few comments and questions I received over the past months.
The tutorial can be found in the packaging-tutorial package (PDF files are in /usr/share/doc/packaging-tutorial/), or on www.debian.org (see links above).
January 1st, 2012 by lucas
> If Ubuntu 12.04 if a LTS release, and Ruby 1.8.7 goes out of support in June of
> 2013, then why is the default still 1.8.7?
> Ruby 1.9.2 was released in 2010. Ruby 1.9.3 was released in October of this year.
First, there’s almost nobody in the Ubuntu development community doing any Ruby work. Packages are just imported from Debian, and Ubuntu follows what is done on the Debian side.
In the Debian/Ruby team, we are currently transitioning to a new packaging helper (gem2deb) that makes it much
easier to support several Ruby versions. Once this will be done, switching to 1.9.3 by default will be very easy. We already provide a way for the sysadmin to change the default on a system.
Now, doing that transition takes time, and we could have used *your* help (and could still use it). We are still quite on time to do it for Debian wheezy, but it sounds very hard to do it for Ubuntu 12.04 unless someone from Ubuntu steps
up to help.
November 6th, 2011 by lucas
I must admit that I’ve never been a big fan of the dash as /bin/sh change. I have three main problems with the switch:
POSIX compliance as an argument
Complying to standards is a really good thing. But when everybody is ignoring the standard because they want the comfort of newer features, maybe it’s a sign that the standard should be updated to include those newer features. Most of the bashims used everywhere in scripts were signifiant improvements, like the ability to write:
cp short1/path1/very/long/common/path/to/a/file short2/path2/very/long/common/path/to/a/file
The option to improve bash was not fully explored
We started with the premise that bash is bloated, slow, and cannot be improved. Maybe you can help me with that, but I could only find a few simplistic benchmarks comparing dash and bash, and I could not find any analysis of why bash is slow, and why it cannot be improved.
One of the obvious problems is that bash is also an interactive shell, and is linked to ncurses and terminfo, which increases the startup time. But have we investigated the possibility to have a /bin/bash-non-interactive binary that would not be linked to ncurses?
The change was brought to users
While it is OK for Debian (or Ubuntu, in that case, since that change was done in Ubuntu first) to force its developers to use POSIX-compliant scripts, the switch could have been made only to Debian-created scripts (by switching them from a /bin/sh shebang to a /bin/dash shebang, for example). I have trouble justifying that this change was forced on users as well.
Next: linker changes
… and we are doing it again. A set of linker changes (see also the Ubuntu page) was already done in Ubuntu, and is very likely to be done in Debian as well. This switch requires deep changes in some buildsystems (it requires ordering of libraries and forbids indirect dependencies), and is rather painful (it was reverted before the Ubuntu 11.04 release because it was not possible to fix all the packages during the natty release cycle, but is done in the 11.10 release). Of course, there are justifications for this change. But I’m not sure that it’s worth all the trouble created for users.
September 22nd, 2011 by lucas
Windows has a feature called Dynamic Disks, which makes it ignore the DOS partition table and manage partitions its own ways. gparted is supposed to detect that, and warn the user that it can’t deal with that. The solution is to reboot in Windows, switch the partition back to “Basic disks”, and then proceed with installing Debian (or Ubuntu).
However, a student of mine had an interesting experience because gparted did not detect the dynamic disk stuff and did not warn about it, so the student went ahead with installing Debian squeeze. After reboot in Windows, Windows sees the partition as “invalid”. The same happens with Ubuntu 11.04 (not really surprising).
I’m surprised I did not find much information about that issue. Does someone has more info ?
The student was using two disks in his laptop, and only the second one (not the one used to boot windows) was using “dynamic disks”.
September 13th, 2011 by lucas
I must admit that I’m a bit lost about the conclusions of the Don’t fear the fsync()! (lf.org down; google cache) debate. My understanding was that using
fsync() was the right thing to do when we cared about data being written to disk.
tar, I usually care about my data being written to the disk, so why don’t they use
fsync()? Shouldn’t they?
August 1st, 2011 by lucas
So, the fifth Debconf I’ve had the chance to attend is clearly over now. It was great, and all organizers (and sponsors) really deserve huge “thank you” for making this event so successful. I’m already looking forward to next year’s edition.
Debconf has been very productive for me. I chaired 4 sessions:
- the usual Quality Assurance BOF, which was a bit depressing: even if work gets done, the QA “team” doesn’t really feel like a “team”. Maybe that’s because a QA “team” is not needed, and we should instead split it into smaller teams focusing on subsets of the QA work (archive-wide testing, QA services, etc.)?
- a BOF on Ruby, where I demoed the work we have been doing around gem2deb, our new dh-based packaging helper [Ruby team website]
- a tutorial on archive testing, hoping to get more people involved in tests such as archive rebuilds. If you are interested in helping with reporting bugs, please drop me an email [doc about archive tests on wiki.d.o]
- a BOF on finding a ‘standard’ Git workflows for packaging teams. More work is needed on this, but it looks like a good start. [thread on -devel@, wiki page, BOF notes]
I’ve also made numerous uploads of Ruby-related packages, and reduced my backlog on UDD to a reasonable level. I even managed to make a developers-reference upload, integrating all the pending patches from the BTS.
In other news, I’m very excited about the recent progress on expo.debian.net (a mentors.debian.net replacement), which could help streamline our sponsorship process.
July 4th, 2011 by lucas
Next week, I’ll head to Strasbourg for Rencontres Mondiales du Logiciel Libre 2011. On monday morning, I’ll be giving my Debian Packaging Tutorial for the second time. Let’s hope it goes well and I can recruit some future DDs!
Then, at the end of July, I’ll attend Debconf again. Unfortunately, I won’t be able to participate in Debcamp this year, but I look forward to a full week of talks and exciting discussions. There, I’ll be chairing two sessions about Ruby in Debian and Quality Assurance.
May 27th, 2011 by lucas
This week I finally used my packaging tutorial slides myself, a few days after Phil Hands used them at a UKUUG event.
The tutorial is now available as a Debian package (in Debian unstable) named
packaging-tutorial, and I’ve talked to debian-doc@ about linking it from appropriate places. It’s also hosted on collab-maint now, so don’t hesitate to contribute.
May 27th, 2011 by lucas
(this is a copy of an mail sent to ruby-core@ and ruby-talk@)
Since the beginning of 2011, the Debian Ruby team has been working on several big changes. Those changes all are available in Debian unstable, some of them are also available in Debian testing, and they should all be available in the next Debian and Ubuntu releases.
I think that it addresses most of the reasonable concerns about Ruby in Debian.
Using alternatives to switch between Ruby implementations
The alternatives system is now used to manage the “ruby” symlink and the other related symlinks, making it easy to switch between Ruby implementations (only Ruby 1.8 and 1.9.X at the moment) (see this mail for details). The default choice for Ruby is still 1.8, but this change will make it easy for us to make a switch to 1.9.X by default (likely by the release of Debian wheezy).
Installing gems executables to /usr/local/bin
Rubygems (both as a standalone package, and as shipped with Ruby 1.9.X) now install executables to /usr/local/bin. (The other files still get installed under /var, see this mail)
Enabling gem update –system
gem update –system has been re-enabled. Since upgrading rubygems to a version that may not have been properly tested with the rest of the Debian system may cause issues in the user’s system, there’s a big warning about that. The user can confirm and upgrade rubygems anyway by defining an environment variable.
New gem2deb packaging helper
There’s a new packaging helper, named gem2deb, that makes it very easy to generate Debian source packages from Rubygems. We are in the process of migrating all ruby libraries packaged in Debian to that new helper. It will take some time, though (help is welcomed).
Transition status: http://pkg-ruby-extras.alioth.debian.org/wheezy/
One big benefit of the switch to gem2deb for the Ruby community is that, in the process, we are enabling test suites at build time for each package and each Ruby implementation. This should make it easy to detect regressions in new interpreter versions.
We will switch to Ruby 1.9.3 ASAP (probably when it is branched off trunk, with a package first in Debian experimental). Since the Ruby compatibility version issue is likely to stay around, we will re-evaluate how we are dealing with it (to avoid the ruby1.9.1 package <=> ruby -v = 1.9.2 problem that confuses many users). This is likely by switching the package name to ruby1.9.3 (keep a ruby1.9.1 package for compatibility). The package containing the shared library will stay libruby1.9.1.