Automatically watching for updates on web pages ?

Once in a while, I come upon a web page that :

  1. Doesn’t offer an RSS/Atom feed.
  2. Doesn’t change very often.
  3. I would like to be warned when it’s updated.

I would like to be automatically warned when such pages change. websec does this :

Description: Web Secretary – Web page monitoring software
A visual Web page monitoring software. However, it goes
beyond the normal functionalities offered by such software. Not only
does it detect changes based on content analysis (instead of date/time
stamp or simple textual comparison), it will email the changed page to
you with the new content highlighted.

But :

  1. It sends emails. Generating an RSS feed would be much better.
  2. It sends HTML emails. OK, you can use AsciiMarker to view the changes with a text MUA, but still…

Anybody knows of another piece of software I could use ?

Who is using Debian’s and Ubuntu’s development versions ?

Scott J. Remnant claimed that it’s the development community that use Debian testing or unstable. This doesn’t match my personal experience (I know many people running Debian testing/unstable who aren’t interested in Debian development), however I needed a better way to check this than just “my personal experience”. So I tried to answer the following question :

Are normal users using Ubuntu edgy and Debian testing/unstable, or do they stick with Ubuntu dapper and Debian stable ?

Simplified summary: (read below for details)

Ubuntu users:
…Ubuntu dapper (stable): 96%
…Ubuntu edgy (development): 4%
Debian users:
…Debian sarge (stable): 24%
…Debian testing/unstable (development): 76%

Detailed process:

First, I did a poll on #ubuntu@irc.freenode.net using a CTCP VERSION query, and looked at people using Xchat and Ubuntu (Ubuntu is displayed in CTCP VERSION replies). I got 288 replies from Xchat users, 197 of them were using the Xchat package from Ubuntu. 189 (96%) of them are using version 2.6.1 (the version in Dapper), 8 (4%) of them are using versions 2.6.4 or 2.6.6.

Then, I did the same poll on #debian@irc.oftc.net. I got 85 replies from Xchat users, and filtered out 15 of them because they weren’t from Debian users for obvious reasons. 17 users (24%) are using versions 2.4.0 or 2.4.1 (Debian sarge), 53 users (76%) are using versions 2.4.3, 2.4.4, 2.4.5, 2.6.1 and 2.6.4 (Debian testing/unstable – v.2.6.4, the current version in testing/unstable, has 41 users on its own). Statistically speaking, this isn’t very good, so I compared those results with another source of information : Debian popularity contest. 3002 submitters are using v.1.28 of the popularity-contest package (version in sarge) while 9682 submitters are using v.1.33 (version currently in Debian testing/unstable). That accounts for respectively 24% and 76% ! Same as above ! (OK, this is not so good, because I haven’t considered the intermediate versions here, while I did above)
Of course, those numbers are huge approximations, and are subject to a lot of hypotheses and questions, amongst them:

  • Are people on #ubuntu and #debian comparable ?
  • Are Xchat users or popcon participants representative enough ?
  • What about users of other IRC clients ? (I can’t do it with irssi, since its version didn’t change between dapper and edgy)
  • What about people who change their CTCP VERSION reply ? Or disable it ?
  • This only shows the system that people use to do IRC.
  • Xchat users obviously run X on the system they are using. Irssi users don’t necessarly do, and I got slightly different results with #debian and irssi : 55 users of v.0.8.10 (Debian testing/unstable), versus 30 users of v.0.8.9 (Debian stable). That makes 65%/35% instead of 76%/24%.
  • This is done in the middle of the edgy dev cycle, and near the end of the etch dev cycle. This could change over time.

I didn’t run the same poll on #debian-devel and #ubuntu-devel because I received some unpleasant comments already, and don’t want to run into angry Debian or Ubuntu developers (I’m still waiting for DAM approval after all :-)

Ubuntu or Debian, which one has the newest software ?

It is a common belief that Ubuntu provides newer software than Debian. This is of course true when Ubuntu is compared to Debian stable. This is also true for specific software packages, like Gnome. But how does it compare in general ?

At the time of the Dapper release, here is how Ubuntu Dapper and Debian Unstable compared wrt source package versions :

In Ubuntu, but not in Debian: 1795
In Debian, but not in Ubuntu: 588
Exact same version: 4361
Same upstream version: 2963
… and same Debian version, but Ubuntu has local changes: 731
… “Debian version” newer in Debian: 2197
… “Debian version” newer in Ubuntu: 35
Different upstream version: 2166
… newer in Debian: 1955
… newer in Ubuntu: 211

So Debian easily wins. But this comparison is unfair, because Debian unstable is a moving target : of course, it’s newer than Ubuntu ! What about Debian testing ?

In Ubuntu, but not in Debian: 2137
In Debian, but not in Ubuntu: 514
Exact same version: 4529
Same upstream version: 2652
… and same Debian version, but Ubuntu has local changes: 775
… newer in Debian: 1816
… newer in Ubuntu: 61
Different upstream version: 1967
… newer in Debian: 1666
… newer in Ubuntu: 301

Yes, Debian still wins. Okay, but Ubuntu Dapper had a particularly long release cycle. What about Ubuntu Breezy, and Debian Testing at the time of the Breezy release ?

In Ubuntu, but not in Debian: 2462
In Debian, but not in Ubuntu: 384
Exact same version: 4586
Same upstream version: 2253
… and same Debian version, but Ubuntu has local changes: 981
… newer in Debian: 1172
… newer in Ubuntu: 100
Different upstream version: 1375
… newer in Debian: 1020
… newer in Ubuntu: 355

Debian testing wins again.

Notes:

  • Statistics might be biased because of source package name changes.
  • Ubuntu doesn’t automatically remove packages which were removed from Debian, which partially explains the high number of packages in Ubuntu but not in Debian.
  • OK, there’s no Debian unstable/Ubuntu edgy comparison. I believe that not many people use Ubuntu development versions (except a few weeks before release), and that Ubuntu dev versions are usually not considered usable. OTOH, many people use Debian testing or Debian unstable.
  • The quick and dirty script used to generate the stats is available.

Update:

Yes, Scott, I am comparing stable releases on one side, with development versions on the other side. However, I think that it is widely known that Debian testing is usable outside of the Debian development community. I know many people using Debian testing who aren’t Debian developers. And many Ubuntu users, if they weren’t using Ubuntu, but Debian, would probably use Debian testing, not stable.

Regarding Ubuntu Edgy, its use is currently strictly limited to the Ubuntu development community, and I don’t know anybody not interested in Ubuntu development running Ubuntu Edgy (that’s also because Ubuntu is released every 6 months, not 18 months, of course). That’s why it’s irrelevant to compare with it.
Also, after a quick poll on IRC, I discovered that many Ubuntu users thought that Ubuntu stable releases had newer software than Debian Testing (that was my exact question). The only point of this blog entry is to prove them wrong.

Update 2 : I’ll be on holidays for the next few days, so comments are put in the moderation queue until I come back.

Good questions to ask yourself when working inside teams

Many people agree that teams are good : doing team maintenance of packages is often seen as A Good Thing inside Debian, and Ubuntu enforces it by switching to the everything is team-maintained philosophy. However, teams are subject to complex group dynamics, and raise interesting issues that you don’t encounter when working alone. It is important to ask yourself those few questions

Dilution of knowledge: Team members tend to individually know less about specific packages. Does the sum of all team members really know more than an individual developer would know about a specific package ?

Responsibility: Are there some clear responsibilities in your team ? Are people feeling responsible for packages ? Not having clear responsibilities is dangerous, because a package could become sort-of-orphaned inside the team, because nobody would consider “his duty” to work on it.

Hierarchy inside teams: Is there a clear hierarchy inside your team ? If there isn’t (which could be OK), are you sure there isn’t an unofficial hierarchy that built up with time, causing people to wait for an unofficial leader’s decision, while this leader might not even be aware that he is that leader ?

Scalability: How much can a team scale ? How does *your* team scale ? How many team members are really active ?

External interface: When dealing with the outside world (read: upstream), it’s easier to have a single point of communication. How does your team’s external interface look like ? Remember that teams’ external interfaces can easily become much more complex for upstream developers to deal with.

There are no magic way to build a team that work, but it’s important to look at your team with a critical eye and try to improve its inner workings.