dd + /dev/random fun

From Adrien Lebre, who seems to be having fun working on kDFS:

$ dd if=/dev/random of=file bs=1024 count=2
0+2 records in
0+2 records out
256 bytes (256 B) copied, 0.000705717 s, 363 kB/s
$ ls -l file
-rw-r--r-- 1 lucas lucas 256 2008-02-08 17:53 file

Can be easily explained using strace:

open("/dev/random", O_RDONLY|O_LARGEFILE) = 0
[..]
open("file", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 1
[..]
read(0, "p(\361\241,\360-\330~M\245y\10=\274U\201c\207\v\336a\273"..., 1024) = 128
write(1, "p(\361\241,\360-\330~M\245y\10=\274U\201c\207\v\336a\273"..., 128) = 128
read(0, "\362,LW5l.?\242\22\252\223\206\375\10\326\335\316\374\372"..., 1024) = 128
write(1, "\362,LW5l.?\242\22\252\223\206\375\10\326\335\316\374\372"..., 128) = 128
close(0)                                = 0
close(1)                                = 0

Re: a problem with tools

Joey has a problem with tools used over dpkg-dev’s core tools to help maintainers maintain their debian packages.

I was wondering about the adoption of those tools, so I simply grepped the logs from my last rebuild for “^Setting up $TOOL “, which is more accurate than looking at build-deps, since someteam-pkg-tools could depend on cdbs, and hide cdbs usage.

Results, with lists of packages “affected”:

Tool Packages
debhelper 11434 (95.8%)
cdbs 2665 (22.3%)
dpatch 1714 (14.4%)
quilt 935 (7.8%)
dbs 57 (0.47%)
yada 32 (0.27%)
debmake 2 (0.017%)

(if you want me to check the logs for another tool, just tell me)

My personal opinion on this is that tools that make sense are slowly getting adopted (that probably includes debhelper, cdbs, dpatch and quilt). The main difference between debhelper and the other tools is that the other tools don’t try to solve a problem for everybody. cdbs perfectly makes sense for some packages: the simple ones, where you only have to modify 4-5 lines in the output from dh_make to build properly.

Patch systems are also a good way to track changes made to upstream source, and probably encourage giving back patches to upstream. Note that my stats don’t include packages using cdbs’ simple-patchsys, so there are probably really close to 1/4 or 1/3 of our packages using some sort of patch system. How course, having simple-patchsys, dpatch and quilt sounds a bit stupid. But last time I checked, nobody took the time to blog a comparison of those tools, saying why {quilt,dpatch} is better than {simple-patchsys,dpatch,quilt}. Also, simple-patchsys probably makes a good-enough job for its users, so they don’t see the point in changing.

Of course, it’s easier to bury your head in the sand, and ignore the fact that those tools solve real problems for so many people and packages, and make their lives easier. :-)

Fixing build failures with dash is cool.

The bad thing with fixing build failures with dash is that there are still a lot of open bugs. (Remember, the 0-day NMU policy applies to those, so it’s a good opportunity to improve your NMU karma. And I will sponsor your NMUs if you can’t upload!).

The good thing is that you sometimes run into funny code, like:
clean:
    dh_testdir
    dh_testroot
    pushd docs ; $(MAKE) distclean || true ; popd

Since pushd/popd is a bashism, the Ubuntu patch changed this to:
clean:
    dh_testdir
    dh_testroot
    cd docs ; $(MAKE) distclean || true ; cd $(CURDIR)

Which works, but is … interesting? :-)

Help us get rid of dash build failures in Debian!

Following my rebuild of all packages using /bin/dash as /bin/sh, we now have a nice list of bugs.

Those bugs are cool:

  • The relaxed NMU rules apply to them, since they are part of the dash release goal. Which means that they can all be fixed using 0-day NMUs.
  • Most of them are really easy to fix. (for many of them, a patch is already provided by Ubuntu)

So, if you want to get involved in Debian development, try to submit NMU patches for those bugs. If you want me to sponsor the upload, please Cc me when you submit the patch.

And even if you don’t care about Debian, but only about Ubuntu (which is fondamentally wrong, as everybody knows that Ubuntu is based on Debian), fixing these bugs also helps Ubuntu a lot: all those bugs hurt Ubuntu, since Ubuntu uses dash as /bin/sh by default, and, even if the packages in Ubuntu have been patched, it’s still causing additional work every time a new package is uploaded to Debian, and has to imported in Ubuntu.

If you have questions with the process, please ask them using the comments below.

Notes:
I use the following process on those bugs, and it’s probably a good idea that you do the same if you intent to submit patches:

  • build the package with a chroot using bash
  • build the package with a chroot using dash (you can just use a different pbuilder tarball for that). Check that the problem can be reproduced.
  • fix the package
  • build the new package with a chroot using bash
  • build the new package with a chroot using dash
  • compare the built packages using debdiff for old_package_using_bash, new_package_using_bash, and new_package_using_dash. If the content doesn’t match, we have a problem.

If you submit NMU patches, please also have a look at the other bugs in the package, or at the lintian output. I don’t mind sponsoring small fixes for non-cosmetic/non-wishlist stuff at the same time.

Rebuilding the archive with dash as /bin/sh

I rebuilt all Debian packages twice. The first time with bash as /bin/sh, and the second time with dash as /bin/sh.

About 120 packages built fine with bash, but failed to build with dash. I filed all the bugs that weren’t filed already.

Then I debdiff’d the resulting binary packages, and found about 40 packages that built fine with bash and dash, but produced different binary packages (mostly files missing/being added)!

All the bugs (including those that were already filed) have been usertagged. Most of those bugs are quite easy to reproduce and fix, so, if you are bored, send patches!

I was a bit surprised by the high number of problems I found, since Ubuntu has been using dash as /bin/sh for more than a year, even on their buildds. Some packages have been fixed in Ubuntu, but the Debian maintainers didn’t include the fix. But many packages were simply broken in Ubuntu.

Now I’ll try to find time to work on the Build Daemon From Hell idea, and on rebuilds inside qemu.

Ubuntu giving back to Debian: facts and numbers!

I’ve always been annoyed by the discussions about “is Ubuntu really giving back to Debian?”. Debian Developers usually don’t see a lot of “giving back”, and Ubuntu Developers complain about Debian Developers ignoring their bug reports and patches.

So, a few months ago, I proposed that Ubuntu developers use a usertag when they report bugs to the Debian BTS, so they can be tracked.

Results are available:

Comments:

  • It’s really good to see Ubuntu developers reporting bugs and contributing patches to Debian!
  • … But more bugs (and patches) would be better, of course. Let’s continue the good work!
  • Many patches are applied very fast in Debian (as usual), but in some cases, the patches are ignored (as usual, too). It would be great if Debian Developers could treat those bugs as higher priority, since it makes life easier on the Ubuntu side (less difference means less work)

If you are an Ubuntu Developer, read wiki.u.c/Bugs/Debian/Usertagging for the details on how to tag the bugs you file. Note that the submittodebian script in the ubuntu-dev-tools package already sets the usertags.

Is it hard for new contributors to help Debian? Can we improve things a bit?

Debian could use more manpower, but is it actually easy for new contributors who would like to help to do useful things, and get the impression that they actually improved Debian?

There are several problems here:

  • Find bugs that are not too hard to fix. I don’t know about your packages, but in mine, there are some bugs that are easy to fix, and it’s just a matter of finding the time. If someone provided a patch, I would be very happy and apply it or provide feedback as fast as possible.
  • Find packages with active maintainers. It is no secret that if you choose the wrong package, your patch could very well sleep on the BTS for years.
  • Find bugs that actually matter. It’s not strictly a requirement. But fixing bugs in packages that should probably be removed from Debian instead is not really rewarding.

I think that other projects are better than us at providing directions to newcomers. Gnome has the whole Gnome Love thing, for example.

I’m thinking about building lists of “interesting” bugs. That would include RC bugs without patches, of course, but could also include a list of bugs based on an usertag that would mean “This bug is suitable for new contributors (it’s not too hard, and you could learn things fixing it). If you want to work on this bug, I’ll provide help and feedback and integrate your patch ASAP.

(This is very different from the “help” tag, which usually means “I couldn’t fix this, it’s too hard, I need help.“. It’s a bad idea to direct newcomers to bugs that are too hard for the package’s maintainer!)

Additionally, a nice feature would be to be able to filter the list of bugs with the packages that the user has installed.

  • What do you think?
  • As someone willing to contribute to Debian, do you often find it hard to find stuff you could do?
  • As a maintainer, do you have a lot of bugs that could use such a tag? Would you use it?

Comments are open, please us them ;)

SC07, MIT Fab labs, and empowering people

This morning’s SuperComputing’07 keynote was a talk by Neil Gershenfeld, director of the Center for Bits and Atoms at the MIT. He mentioned the MIT Fab Labs, a project to bring “Personal Fabrication” to people around the around.

At our own level, that’s something I find very exciting with Free Software: it empowers people to do things with their computers that they couldn’t do if they were using proprietary software, simply because proprietary software is built to fit most people’s needs, not very specific goals some people could have. (and this is similar to the Long Tail stuff, in some ways)

Also, if you are at SC07 and reading this, feel free to ping me! I haven’t seen a lot of Debian/Ubuntuers here, even if the Free/Proprietary software ratio is very high.

Better mentors.debian.net/REVU to improve our sponsorship workflow?

I recently sponsored several uploads, and was asked to sponsor even more uploads, and that got me thinking about our sponsorship workflow. It’s a clear bottleneck in Debian, and discourages many new contributors, which obviously sucks.

It’s important to note that the same problems exist in Ubuntu (their equivalent to mentors.debian.net is named REVU).

The best way to improve the process would be to have packages of better quality when a DD first look at them. They would be more likely to be uploaded right away, which frees time for other packages. I think that there’s a lot of room for improvement in the current mentors.debian.net implementation. Here is a small list of features I would like to see.

  • Integration of some QA tests in mentors, as soon as the package is uploaded:
    • does the package build cleanly?
    • piuparts test?
    • lintian/linda checks?
  • Better list of packages awaiting sponsors, with info including:
    • does the package fixes bugs (number of bugs fixed per severity)?
    • is that package already in Debian?
    • is that package a new upstream version?
    • popcon score
    • how long has the package been waiting?

    This would allow potential sponsors to prioritize requests.

  • A commenting system, for each package, so comments for rejected packages are not lost, and the next potential sponsor can double-check
  • A way for sponsors to mark some sponsorees as “friends”, so it’s easy to find all the requests from people I “trust” (for some definitions of “trust” ;)
  • Maybe, a scoring system, where providing good comments on other’s packages would make you win “karma points”, and improve your classification, which could later be used by sponsors to choose what they are going to sponsor next.

The good thing with this whole list of features is that everybody can help. So, if you are looking for a sponsor and want to help solve this problem, start coding now ;) And if you need me to create nicenameforyourservice.debian.net, just ping me. There’s probably some code to steal from svnbuildstat.debian.net, so contacting its developers would be a good idea.

Re: Giving credit where due?

Christian Perrier is wondering why the official announcement about the Gutsy release does not even contain the word “Debian”.

It’s not new: Debian is virtually nonexistent in Ubuntu’s communication. It seems that the last Ubuntu release to acknowledge its Debian origins was Dapper (June 2006), maybe because that was the “Long Term Support” release.

The fact that there’s no “Ubuntu is based on Debian” paragraph on www.ubuntu.com was raised during Debconf, and it was supposed to get fixed, but it seems that it didn’t happen for some reason (there was such a paragraph before the website redesign).

In other news, I’ve been trying to install Ubuntu Gutsy inside qemu, but it fails miserably while booting the installer. I removed the “quiet” and “splash” options from the kernel cmdline, and discovered that after trying to “mount the root filesystem”, I get dropped into busybox with no error message to google for. Feisty fails as well, but Dapper boots fine. So much for the Ubuntu is an ancient African word, meaning “I can’t install Debian” joke!