My vote:


> [ 1 ] Choice 1: F: Focus on systemd
> [ 2 ] Choice 2: B: Systemd but we support exploring alternatives
> [ 3 ] Choice 3: A: Support for multiple init systems is Important
> [ 4 ] Choice 4: D: Support non-systemd systems, without blocking progress
> [ 4 ] Choice 5: H: Support portability, without blocking progress
> [ 5 ] Choice 8: Further Discussion
> [ 6 ] Choice 6: E: Support for multiple init systems is Required
> [ 7 ] Choice 7: G: Support portability and multiple implementations


Rationale:

At this point, I don’t think that it is useful anymore for Debian to spend energy on supporting several init systems. I believe that experimentation is useful, that Debian should support it, but that it does not need to happen inside Debian. Interested people can work on derivative distributions, or even just maintain a small set of packages installable on top of Debian that will add support for alternative init systems where needed.

This is a slightly edited copy of the mail sent to debian-devel@.

TL;DR: see https://trends.debian.net and https://trends.debian.net/#smells.

Following this blog post I did some work on setting up a proper framework to graph historical trends about Debian packaging practices. The result is now available at https://trends.debian.net/, and I’m confident that I will be able to update this on a regular basis (every few months).

Additionally (and much more controversially I guess :-) ) I also added an analysis of “package smells”, such as “not using dh”, “not using a recent debhelper compat level”, “not using a 3.0 source format”, etc. I understand that in some cases there might be good reasons to keep those “smells”, but I find it valuable to have them presented in a more actionable way to fix the cases that should be fixed. So there’s a list of smells, sorted by maintainer/uploader.

Given that Debian is currently frozen to prepare the buster release, this is a bad time to start fixing those smells, but I will send a reminder to debian-devel@ once buster is released. (It’s interesting to see how the number of smells plateaued during previous freezes).

If you are still running jessie you probably noticed that the jessie-updates and jessie-backports suites have been removed from mirrors, because you got those error messages:

W: Failed to fetch http://ftp.debian.org/debian/dists/jessie-updates/main/binary-amd64/Packages 404 Not Found [IP: 130.89.148.12 80]
W: Failed to fetch http://ftp.debian.org/debian/dists/jessie-backports/main/binary-amd64/Packages 404 Not Found [IP: 130.89.148.12 80]

I was not involved in that decision (which was made by the FTP masters team), but since there is some confusion around it, I will try to give my understanding of the resulting issues.

The typical /etc/apt/sources.list file for a jessie system with backports enabled is:

deb http://ftp.debian.org/debian jessie main
deb-src http://ftp.debian.org/debian jessie main

deb http://security.debian.org/debian-security jessie/updates main
deb-src http://security.debian.org/debian-security jessie/updates main

deb http://ftp.debian.org/debian jessie-updates main
deb-src http://ftp.debian.org/debian jessie-updates main

deb http://ftp.debian.org/debian/ jessie-backports main contrib non-free
deb-src http://ftp.debian.org/debian/ jessie-backports main contrib non-free

Debian packages are distributed using suites (which can be understood as channels). The global picture looks like this:

(This is slide 42 of the Debian Packaging Tutorial.)

deb http://deb.debian.org/debian jessie main is the easy one. It contains the bulk of packages. It is initialized by copying the content of the testing suite when a new stable release happens, approximately every two years. It is then updated from stable-new (an internal suite) when stable point releases happen (see below).

deb http://security.debian.org/debian-security jessie/updates main is the security suite in the figure above. It is used by the Debian security team to provide security updates. They are announced on the debian-security-announce mailing list.

deb http://ftp.debian.org/debian jessie-updates main (stable-updates above) is a suite used to distribute important updates that are unrelated to security, and that cannot wait the next stable point release. They are announced on the debian-stable-announce mailing list. Interestingly, a large proportion of those updates are related to changes to daylight-saving-time rules that are sometimes made very late by some countries.

stable point releases happen every few months (see for example the Debian 8.11 stable point release). They consist in updating the stable suite by copying important updates that were submitted to stable-proposed-updates. Security updates are also included.

backports follow an entirely different path. They are new versions of packages, based on the version currently in the testing suite. See the backports team website.

So, what happened?

In June 2018…

Debian 8.11 (in June 2018) was the final update for Debian 8. As stated in its announcement:

After this point release, Debian's Security and Release Teams will no
longer be producing updates for Debian 8. Users wishing to continue to
receive security support should upgrade to Debian 9, or see
https://wiki.debian.org/LTS for details about the subset of
architectures and packages covered by the Long Term Support project.

In other words: jessie and jessie-updates won’t receive any update. The only updates will be through the security suite, by the Debian Long Term Support project.

At about the same time (I think – I could not find an announcement — Update: announcement), the maintenance of backports for jessie was also stopped. Which makes sense, because the backports team provides backports for the current release, and stretch was released in June 2017.

In March 2019…

The FTP masters team decided to remove the jessie-updates and jessie-backports suite from the mirrors. This was announced on debian-devel-announce, resulting in the errors quoted above.

How to solve this?

For the jessie-updates suite, you can simply remove it from your /etc/apt/sources.list. It is useless, because all packages that were in jessie-updates were merged into jessie when Debian 8.11 was released.

The jessie-backports suite was archived on archive.debian.org, so you can use:

deb http://archive.debian.org/debian/ jessie-backports main contrib non-free
deb-src http://archive.debian.org/debian/ jessie-backports main contrib non-free

But then you will run into another issue:

E: Release file for http://archive.debian.org/debian/dists/jessie-backports/InRelease is expired (invalid since 36d 1h 9min 51s). Updates for this repository will not be applied.

Unfortunately, with the APT version in jessie, this cannot be ignored on a per source basis (it can with the APT version from stretch, using the deb [check-valid-until=no] ... syntax). So you need to disable this check globally, using:

echo 'Acquire::Check-Valid-Until no;' > /etc/apt/apt.conf.d/99no-check-valid-until

After that, apt-get update just works.

(There are some discussions about resurrecting the jessie-updates suite to avoid the above errors, but it is probably getting less and less useful as time passes.)

It has been raised in various discussions how much it’s difficult to make large-scale changes in Debian.

I think that one part of the problem is that we are not very good at tracking those large-scale changes, and I’d like to change that. A long time ago, I did some graphs about Debian (first in 2011, then in 2013, then again in 2015). An example from 2015 is given below, showing the market share of packaging helpers.

Those were generated using a custom script. Since then, classification tags were added to lintian, and I’d like to institutionalize that a bit, to make it easier to track more trends in Debian, and maybe motivate people with switching to new packaging standards. This could include stuff like VCS used, salsa migration, debhelper compat levels, patch systems and source formats, but also stuff like systemd unit files vs traditional init scripts, hardening features, etc. The process would look like:

1. Add classification tags to lintian for relevant stuff (maybe starting with being able to regenerate the graphs from 2015).
2. Use lintian to scan all packages on snapshot.debian.org, which stores all packages ever uploaded to Debian (well, since 2005), and generate a dataset
3. Generate nice graphs

Given my limited time available for Debian, I would totally welcome some help. I can probably take care of the second step (I actually did it recently on a subset of packages to check feasibility), but I would need:

• The help of someone with Perl knowledge, willing to modify lintian to add additional classification tags. There’s no need to be a Debian Developer, and lintian has an extensive test suite, that should make it quite fun to hack on. The code could either be integrated in lintian, or live in a lintian fork that would only be used to generate this data.
• Ideally (but that’s less important at this stage), the help of someone with web skills to generate a nice website.

Let me know if you are interested.

Michael Stapelberg writes about his frustrations with Debian, resulting in him reducing his involvement in the project. That’s sad: over the years, Michael has made a lot of great contributions to Debian, addressing hard problems in interesting, disruptive ways.

He makes a lot of good points about Debian, with which I’m generally in agreement. An interesting exercise would be to rank those issues: what are, today, the biggest issues to solve in Debian? I’m nowadays not following Debian closely enough to be able to do that exercise, but I would love to read others’ thoughts (bonus points if it’s in a DPL platform, given that it seems that we have a pretty quiet DPL election this year!)

Most of Michael’s points are about the need for modernization of Debian’s infrastructure and workflows, and I agree that it’s sad that we have made little progress in that area over the last decade. And I think that it’s important to realize that providing alternatives to developers have a cost, and that when a large proportion of developers or packages have switched to doing something (using git, using dh, not using 1.0-based patch systems such as dpatch, …), there are huge advantages with standardizing and pushing this on everybody.

There are a few reasons why this is harder than it sounds, though.

First, there’s Debian culture of stability and technical excellence. “Above all, do not harm” could also apply to the mindset of many Debian Developers. On one hand, that’s great, because this focus on not breaking things probably contributes a lot to our ability to produce something that works as well as Debian. But on the other hand, it means that we often seek solutions that limit short-term damage or disruption, but are far from optimal on the long term.
An example is our packaging software stack. I wrote most of the introduction to Debian packaging found in the packaging-tutorial package (which is translated in six languages now), but am still amazed by all the unjustified complexity. We tend to fix problems by adding additional layers of software on top of existing layers, rather than by fixing/refactoring the existing layers. For example, the standard way to package software today is using dh. However, dh stands on dh_* commands (even if it does not call them directly, contrary to what CDBS did), and all the documentation on dh is still structured around those commands: if you want to install an additional file in a package, probably the simplest way to do that is to add it to debian/packagename.install, but this is documented in the manpage for dh_install, which your are not going to actually call because dh abstracts that away for you! I realize that this could be better explained in packaging-tutorial… (patch welcomed)

There’s also the fact that Debian is very large, very diverse, and hard to test. It’s very easy to break things silently in Debian, because many  of our packages are niche packages, or don’t have proper test suites (because not everything can be easily tested automatically). I don’t see how the workflows for large-scale changes that Michael describes could work in Debian without first getting much better at detecting regressions.

Still, there’s a lot of innovation going on inside packaging teams, with the development of language-specific packaging helpers (listed on the AutomaticPackagingTools wiki page). However, this silo-ed organization tends to fragment the expertise of the project about what works and what doesn’t: because packaging teams don’t talk much together, they often solve the same problems in slightly different ways. We probably need more ways to discuss interesting stuff going on in teams, and consolidating what can be shared between teams. The fact that many people have stopped following debian-devel@ nowadays is probably not helping…

The addition of salsa.debian.org is probably the best thing that happened to Debian recently. How much this ends up being used for improving our workflows remain to be seen:

• We could use Gitlab merge requests to track patches, rather than attachments in the BTS. Some tooling to provide an overview of open MRs in various dashboards is probably needed (and unfortunately GitLab’s API is very slow when dealing with large number of projects).
• We could probably have a way to move the package upload to a gitlab-ci job (for example, by committing the signed changes file in a specific branch, similar to what pristine-tar does, but there might be a better way)
• I would love to see a team experiment with a monorepo approach (instead of the “one git repo per package + mr to track them all” approach). For teams with lots of small packages there are probably a lot of things to win with such an organization.

 

Russell Coker wrote about Sysadmin Skills and University Degrees. I couldn’t agree more that a major deficiency in Computer Science degrees is the lack of sysadmin training. It seems like most sysadmins learned most of what they know from experience. It’s very hard to recruit young engineers (freshly out of university) for sysadmin jobs, and the job interviews are often a bit depressing. Sysadmins jobs are also not very popular with this public, probably because university curriculums fail to emphasize what’s exciting about those jobs.

However, I think I disagree rather deeply with Russell’s detailed analysis.

First, Version Control. Well, I think that it’s pretty well covered in university curriculums nowadays. From my point of view, teaching CS in Université de Lorraine (France), mostly in Licence Professionnelle Administration de Systèmes, Réseaux et Applications à base de Logiciels Libres (warning: french), a BSc degree focusing on Linux systems administration, it’s not usual to see student projects with a mandatory use of Git. And it doesn’t seem to be a major problem for students (which always surprises me). However, I wouldn’t rate Version Control as the most important thing that is required for a sysadmin. Similarly Dependencies and Backups are things that should be covered, but probably not as first class citizens.

I think that there are several pillars in the typical sysadmin knowledge.

First and foremost, sysadmins need a good understanding of the inner workings of an operating system. I sometimes feel that many Operating Systems Design courses are a bit too much focused on the “Design” side of things. Yes, it’s useful to understand the low-level mechanisms, and be able to (mentally) recreate an OS from scratch. But it’s also interesting to know how real systems are actually built, and what are the trade-off involved. I very much enjoyed reading Branden Gregg’s Systems Performance: Enterprise and the Cloud because each chapter starts with a great overview of how things are in the real world, with a very good level of detail. Also, addressing OS design from the point of view of performance could be a way to turn those courses into something more attractive for students: many people like to measure, benchmark, optimize things, and it’s quite easy to demonstrate how different designs, or different configurations, make a big difference in terms of performance in the context of OS design. It’s possible to be a sysadmin and ignore, say, the existence of the VFS, but there’s a large class of problems that you will never be able to solve. It can be a good trade-off for a curriculum (e.g. at the BSc level) to decide to ignore most of the low-level stuff, but it’s important to be aware of it.

Students also need to learn how to design a proper infrastructure (that meets requirements in terms of scalability, availability, security, and maybe elasticity). Yes, backups are important. But monitoring is, too. As well as high availability. In order to scale, it’s important to be able to automatize stuff. Russell writes that Sysadmins need some programming skills, but that’s mostly scripting and basic debugging. Well, when you design an infrastructure, or when you use configuration management tools such as Puppet, in some sense, you are programming, and in terms of needs to abstract things, it’s actually similar to doing object-oriented programming, with similar choices (should I use that off-the-shelf puppet module, or re-develop my own? How should everything fit together?). Also, when debugging, it’s often useful to be able to dig into code, understand what the developer was trying to do, and if the expected behavior actually matches what you are seeing. It often results in spending a lot of time to create a one-line fix, and it requires very advanced programming skills. Again, it’s possible to be a sysadmin with only limited software development knowledge, but there’s a large class of things that you are unlikely to address properly.

I think that what makes sysadmins jobs both very interesting and very challenging is that they require a very wide range of knowledge. There’s often the ability to learn about new stuff (much more than in software development jobs). Of course, the difficult question is where to draw the line. What is the sysadmin knowledge that every CS graduate should have, even in curriculums not targeting sysadmin jobs? What is the sysadmin knowledge for a sysadmin BSc degree? for a sysadmin MSc degree?