Removal of jessie-updates and jessie-backports from Debian mirrors

If you are still running jessie you probably noticed that the jessie-updates and jessie-backports suites have been removed from mirrors, because you got those error messages:

W: Failed to fetch http://ftp.debian.org/debian/dists/jessie-updates/main/binary-amd64/Packages 404 Not Found [IP: 130.89.148.12 80]
W: Failed to fetch http://ftp.debian.org/debian/dists/jessie-backports/main/binary-amd64/Packages 404 Not Found [IP: 130.89.148.12 80]

I was not involved in that decision (which was made by the FTP masters team), but since there is some confusion around it, I will try to give my understanding of the resulting issues.

The typical /etc/apt/sources.list file for a jessie system with backports enabled is:

deb http://ftp.debian.org/debian jessie main
deb-src http://ftp.debian.org/debian jessie main

deb http://security.debian.org/debian-security jessie/updates main
deb-src http://security.debian.org/debian-security jessie/updates main

deb http://ftp.debian.org/debian jessie-updates main
deb-src http://ftp.debian.org/debian jessie-updates main

deb http://ftp.debian.org/debian/ jessie-backports main contrib non-free
deb-src http://ftp.debian.org/debian/ jessie-backports main contrib non-free

Debian packages are distributed using suites (which can be understood as channels). The global picture looks like this:

(This is slide 42 of the Debian Packaging Tutorial.)

deb http://deb.debian.org/debian jessie main is the easy one. It contains the bulk of packages. It is initialized by copying the content of the testing suite when a new stable release happens, approximately every two years. It is then updated from stable-new (an internal suite) when stable point releases happen (see below).

deb http://security.debian.org/debian-security jessie/updates main is the security suite in the figure above. It is used by the Debian security team to provide security updates. They are announced on the debian-security-announce mailing list.

deb http://ftp.debian.org/debian jessie-updates main (stable-updates above) is a suite used to distribute important updates that are unrelated to security, and that cannot wait the next stable point release. They are announced on the debian-stable-announce mailing list. Interestingly, a large proportion of those updates are related to changes to daylight-saving-time rules that are sometimes made very late by some countries.

stable point releases happen every few months (see for example the Debian 8.11 stable point release). They consist in updating the stable suite by copying important updates that were submitted to stable-proposed-updates. Security updates are also included.

backports follow an entirely different path. They are new versions of packages, based on the version currently in the testing suite. See the backports team website.

So, what happened?

In June 2018…

Debian 8.11 (in June 2018) was the final update for Debian 8. As stated in its announcement:

After this point release, Debian's Security and Release Teams will no
longer be producing updates for Debian 8. Users wishing to continue to
receive security support should upgrade to Debian 9, or see
https://wiki.debian.org/LTS for details about the subset of
architectures and packages covered by the Long Term Support project.

In other words: jessie and jessie-updates won’t receive any update. The only updates will be through the security suite, by the Debian Long Term Support project.

At about the same time (I think – I could not find an announcement — Update: announcement), the maintenance of backports for jessie was also stopped. Which makes sense, because the backports team provides backports for the current release, and stretch was released in June 2017.

In March 2019…

The FTP masters team decided to remove the jessie-updates and jessie-backports suite from the mirrors. This was announced on debian-devel-announce, resulting in the errors quoted above.

How to solve this?

For the jessie-updates suite, you can simply remove it from your /etc/apt/sources.list. It is useless, because all packages that were in jessie-updates were merged into jessie when Debian 8.11 was released.

The jessie-backports suite was archived on archive.debian.org, so you can use:

deb http://archive.debian.org/debian/ jessie-backports main contrib non-free
deb-src http://archive.debian.org/debian/ jessie-backports main contrib non-free

But then you will run into another issue:

E: Release file for http://archive.debian.org/debian/dists/jessie-backports/InRelease is expired (invalid since 36d 1h 9min 51s). Updates for this repository will not be applied.

Unfortunately, with the APT version in jessie, this cannot be ignored on a per source basis (it can with the APT version from stretch, using the deb [check-valid-until=no] ... syntax). So you need to disable this check globally, using:

echo 'Acquire::Check-Valid-Until no;' > /etc/apt/apt.conf.d/99no-check-valid-until

After that, apt-get update just works.

(There are some discussions about resurrecting the jessie-updates suite to avoid the above errors, but it is probably getting less and less useful as time passes.)

Call for help: graphing Debian trends

It has been raised in various discussions how much it’s difficult to make large-scale changes in Debian.

I think that one part of the problem is that we are not very good at tracking those large-scale changes, and I’d like to change that. A long time ago, I did some graphs about Debian (first in 2011, then in 2013, then again in 2015). An example from 2015 is given below, showing the market share of packaging helpers.

Those were generated using a custom script. Since then, classification tags were added to lintian, and I’d like to institutionalize that a bit, to make it easier to track more trends in Debian, and maybe motivate people with switching to new packaging standards. This could include stuff like VCS used, salsa migration, debhelper compat levels, patch systems and source formats, but also stuff like systemd unit files vs traditional init scripts, hardening features, etc. The process would look like:

  1. Add classification tags to lintian for relevant stuff (maybe starting with being able to regenerate the graphs from 2015).
  2. Use lintian to scan all packages on snapshot.debian.org, which stores all packages ever uploaded to Debian (well, since 2005), and generate a dataset
  3. Generate nice graphs

Given my limited time available for Debian, I would totally welcome some help. I can probably take care of the second step (I actually did it recently on a subset of packages to check feasibility), but I would need:

  • The help of someone with Perl knowledge, willing to modify lintian to add additional classification tags. There’s no need to be a Debian Developer, and lintian has an extensive test suite, that should make it quite fun to hack on. The code could either be integrated in lintian, or live in a lintian fork that would only be used to generate this data.
  • Ideally (but that’s less important at this stage), the help of someone with web skills to generate a nice website.

Let me know if you are interested.

On Debian frustrations

Michael Stapelberg writes about his frustrations with Debian, resulting in him reducing his involvement in the project. That’s sad: over the years, Michael has made a lot of great contributions to Debian, addressing hard problems in interesting, disruptive ways.

He makes a lot of good points about Debian, with which I’m generally in agreement. An interesting exercise would be to rank those issues: what are, today, the biggest issues to solve in Debian? I’m nowadays not following Debian closely enough to be able to do that exercise, but I would love to read others’ thoughts (bonus points if it’s in a DPL platform, given that it seems that we have a pretty quiet DPL election this year!)

Most of Michael’s points are about the need for modernization of Debian’s infrastructure and workflows, and I agree that it’s sad that we have made little progress in that area over the last decade. And I think that it’s important to realize that providing alternatives to developers have a cost, and that when a large proportion of developers or packages have switched to doing something (using git, using dh, not using 1.0-based patch systems such as dpatch, …), there are huge advantages with standardizing and pushing this on everybody.

There are a few reasons why this is harder than it sounds, though.

First, there’s Debian culture of stability and technical excellence. “Above all, do not harm” could also apply to the mindset of many Debian Developers. On one hand, that’s great, because this focus on not breaking things probably contributes a lot to our ability to produce something that works as well as Debian. But on the other hand, it means that we often seek solutions that limit short-term damage or disruption, but are far from optimal on the long term.
An example is our packaging software stack. I wrote most of the introduction to Debian packaging found in the packaging-tutorial package (which is translated in six languages now), but am still amazed by all the unjustified complexity. We tend to fix problems by adding additional layers of software on top of existing layers, rather than by fixing/refactoring the existing layers. For example, the standard way to package software today is using dh. However, dh stands on dh_* commands (even if it does not call them directly, contrary to what CDBS did), and all the documentation on dh is still structured around those commands: if you want to install an additional file in a package, probably the simplest way to do that is to add it to debian/packagename.install, but this is documented in the manpage for dh_install, which your are not going to actually call because dh abstracts that away for you! I realize that this could be better explained in packaging-tutorial… (patch welcomed)

There’s also the fact that Debian is very large, very diverse, and hard to test. It’s very easy to break things silently in Debian, because many  of our packages are niche packages, or don’t have proper test suites (because not everything can be easily tested automatically). I don’t see how the workflows for large-scale changes that Michael describes could work in Debian without first getting much better at detecting regressions.

Still, there’s a lot of innovation going on inside packaging teams, with the development of language-specific packaging helpers (listed on the AutomaticPackagingTools wiki page). However, this silo-ed organization tends to fragment the expertise of the project about what works and what doesn’t: because packaging teams don’t talk much together, they often solve the same problems in slightly different ways. We probably need more ways to discuss interesting stuff going on in teams, and consolidating what can be shared between teams. The fact that many people have stopped following debian-devel@ nowadays is probably not helping…

The addition of salsa.debian.org is probably the best thing that happened to Debian recently. How much this ends up being used for improving our workflows remain to be seen:

  • We could use Gitlab merge requests to track patches, rather than attachments in the BTS. Some tooling to provide an overview of open MRs in various dashboards is probably needed (and unfortunately GitLab’s API is very slow when dealing with large number of projects).
  • We could probably have a way to move the package upload to a gitlab-ci job (for example, by committing the signed changes file in a specific branch, similar to what pristine-tar does, but there might be a better way)
  • I would love to see a team experiment with a monorepo approach (instead of the “one git repo per package + mr to track them all” approach). For teams with lots of small packages there are probably a lot of things to win with such an organization.