speedtest.net, or how not to do bandwidth tests

April 9th, 2014 by lucas

While trying to debug a bandwidth problem on a 3G connection, I tried speedtest.net, which ranks fairly high when one searches for “bandwidth test” on various search engines. I was getting very strange results, so I started wondering if my ISP might be bandwidth-throttling all traffic except the one from speedtest.net tests. After all, that’s on a 3G network, and another french 3G ISP (SFR) apparently uses Citrix ByteMobile to optimize the QoE by minifying HTML pages and recompressing images on-the-fly (amongst other things).

So, I fired wireshark, and discovered that no, it’s just speedtest being a bit naive. Speedtest uses its own text-based protocol on port 8080. Here is an excerpt of a download speed test:

> HI
< HELLO 2.1 2013-08-14.01
> DOWNLOAD 1000000
< DOWNLOAD JABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFGHIJABCDEFG

Yeah, right: sequences of “ABCDEFGHIJ”. How course, extremely easy to compress, which apparently happens transparently on 3G (or is it PPP? but I tried to disable PPP compression, and it did not see any change).

It’s funny how digging into problems that look promising at first sight often results in big disappointments :-(

self-hosting my calendar, follow-up

February 27th, 2014 by lucas

Following my blog post on the topic, I played a bit with various options.

But let’s explain my use case (which might be quite specific). I need to deal with three main sources of events:

  • the Zimbra instance from my lab. It provides a CalDav interface.
  • the ICS export from my University’s teaching timetable.
  • a calendar for personal stuff. I don’t want to use my lab’s Zimbra for that.

Additionally, I follow some ICS feeds for some colleagues and other events.
I tend to access my calendar mostly on my computer, and sometimes on my N900 phone.

None of the web interfaces I looked at enabled me to (1) manage different calendars hosted on different CalDav servers; (2) subscribe to ICS feeds; (3) provide a CalDav interface to synchronize my phone.

I ended up using a radicale instance for my personal calendar, which was extremely easy to set up. It’s unfortunately a bit slow when there are many events (1600 since 2010 in my case), so I ended up importing only future events, and I will probably have to cleanup from time to time.

I switched to using IceDove with the Lightning add-on to manage all my calendars and ICS feeds. It’s unfortunately slower and less user-friendly than Google Calendar, but I’ll live with it.

On my N900, I used syncevolution to synchronize my various CalDav calendars. It works fine, but understanding how to configure it is rather tricky due to the number of concepts involved (templates, databases, servers, contexts, …). The synchronization is quite slow (several minutes for the 400-events Zimbra calendar), but works.

I also wanted a way to export my calendars to colleagues (both in a “free/busy” version, and in a “full information” version). I quickly hacked something using ruby-agcaldav (which is not packaged in Debian, and required quite a few dependencies, but it was easy to generate packages for all of them using gem2deb — the situation with other languages did not look better).
The resulting script is:


require 'agcaldav'
require 'date'

cal = AgCalDAV::Client.new(:uri => 'LABCALDAVSERVER', :user => 'xx', :password => "xx")
ev = cal.find_events(:start => '2014-02-01', :end => '2200-01-01')

cal = AgCalDAV::Client.new(:uri => 'RADICALESERVER', :user => 'xx', :password => "xx")
ev2 = cal.find_events(:start => '2014-02-01', :end => '2200-01-01')

limit = (Time::now - 7*86400).to_datetime

# create new empty calendars
ncpriv = Icalendar::Calendar.new
ncpub = Icalendar::Calendar.new

(ev + ev2).each do |e|
next if e.end < limit # drop old events to keep the calendar small

# build event for the free/busy calendar
pe = Icalendar::Event.new
pe.start = e.start
pe.end = e.end
pe.klass = "PRIVATE"
pe.transp = e.transp
ncpriv.add(pe)

# build event for the calendar with event information
pube = Icalendar::Event.new
pube.start = e.start
pube.end = e.end
pube.transp = e.transp
if not e.klass == "PRIVATE"
pube.summary = e.summary
pube.location = e.location
end
ncpub.add(pube)
end

# export free/busy calendar
fd = File::new('xx.ics', 'w')
fd.puts ncpriv.to_ical
fd.close

# export calendar with event information
fd = File::new('yy-Zeeh9bie.ics', 'w')
fd.puts ncpub.to_ical
fd.close

So, mostly everything works. The only thing that doesn't is that I haven't found a way to subscribe to an ICS feed on my N900. Any ideas?

self-hosting my calendar

February 25th, 2014 by lucas

I’m trying to self-host my calendar setup, and I must admit that I’m lost between all the different solutions.

My requirements are:

  • (A) manage my own personal calendar using a reasonably modern web interface (probably on my own CalDAV server)
  • (B) display a dozen public ICS calendars in the web interface. Organizing those public calendars in a tree would be great.
  • (C) display several caldav calendars (from two different instances of zimbra), preferably in RW mode
  • (D) provide ICS links with a secret token that allow me to provide a full view of my calendar to some people (except for private events, where I should just be marked ‘busy’)
  • (E) provide ICS links with a secret token that allow me to provide a “busy/available” view of my calendar to some people
  • (F) export something usable on my n900. MFE would be great since that is already known to work.
  • (G) easy to setup (Debian packages available in wheezy or wheezy-backports, especially for the server part)
  • (H) preferably lightweight. I don’t need a full groupware application. I can ignore the other bits if really needed.

It does not seem to be possible to find a single framework doing all of the above. AFAIK:

  • Owncloud does A, D, G
  • Baikal does A. not sure about the rest.
  • For (B), an alternative is to script the download of the ics and then upload it to the CalDAV using cadaver. But that sounds quite low-level for such a trivial use case.
  • I’ve looked at using IceOwl (and Thunderbird+Lightning) with a CalDAV server such as Radicale. That would solve A (using iceowl instead), B, C. But which CalDAV servers support D, E, F ? Radicale does not do any of those, apparently.

What did I miss?

RSS feed available in the Debian Maintainer Dashboard

January 17th, 2014 by lucas

One of my pet projects in Debian is the Debian Maintainer Dashboard. Built on top of UDD, DMD provides a maintainer-centric view to answer the “I have a few hours for Debian, what should I do now?” question (see example).

Christophe Siraut did a lot of great work recently on DMD, rewriting much of the internals. As a result, he also added a RSS feed feature: you can now get notified of new TODO list items by subscribing to that feed.

If you have suggestions or comments, please use the debian-qa@ list (see this thread).

Thanks, Christophe!

talk at Open World Forum

October 7th, 2013 by lucas

I attended Open World Forum last week (thanks to Inria for funding my travel). It was a fantastic opportunity to meet many people, and to watch great talks. If I had to single out just one talk, it would clearly be John Sullivan’s What do you mean you can’t Skype?!.

On Saturday, I delivered a talk presenting the Debian project. It was my first DPL-ish talk to the general public, so it still needs some tuning, but I think it went quite well (slides available). Next opportunity to talk about Debian: LORIA, Nancy, France, 2013-10-17 13:30 (iPAC seminar).

Software for brainstorm / ideas sharing and voting?

September 25th, 2013 by lucas

Some time ago, Ubuntu had Ubuntu Brainstorm, a website where non-developers could submit ideas of improvements, and other people could comment and vote on them. I was wondering if there was existing software to deploy a similar service, e.g. as a plugin to popular forum software. I’ve found ideascale.com, but relying on the Cloud for that is not acceptable for my planned use.

(For clarification: my immediate interest for that is unrelated to Debian work)

DebConf was fantastic (not just the view)

August 21st, 2013 by lucas

So, I’m back from DebConf, which probably translated to the 10 busiest days of my life, but also to one of the best times of my life. The Le Camp venue clearly contributed to this success: having everybody at the same place, but at the same time many opportunities for quiet chat or just enjoying the view, was really a good idea. Everybody who made DebConf possible deserve a huge “thank you”, as well as all attendees: it is really a honor to be a part of such a fantastic community.

Now, let’s go back to daily life, and to my re-filled Debian TODO list!

Debian birthday

August 16th, 2013 by lucas

This upload was the first one of my very first package in Debian. It was sponsored by Dafydd Harries, who I’ve finally met at DebConf13, and got out of NEW on 2005-08-16. Exactly 8 years ago today. I only realized that yesterday evening, and Debian’s birthday feels even more special to me now. Dafydd, it looks like I owe you a lot! :)

DebConf

August 16th, 2013 by lucas

This DebConf is obviously quite special for me, being the DPL. It has really been great so far to talk to meet so many people, and especially to meet so many new Debian contributors or Developers. I’m really happy to see that the next generation is ready! :)

On Sunday, I delivered my Bits from the DPL talk. The video is available, and I’ve finally uploaded the slides (working link here, it seems that Penta ate my slides). I hope you enjoyed it/will enjoy it!

Debian releases used by popcon participants

March 30th, 2013 by lucas

The graph below is generated from popcon submissions. Since they include the version of the popularity-contest package, one can determine the Debian release that was used by the submitter (a new version of the popularity-contest package is generally uploaded just after the release to make that tracking possible).

The graph is similar to the one found on popcon, except that versions newer than the latest stable release are aggregated as “testing/unstable”.

Comments:

  • Popcon submitters might not be representative of Debian users, of course.
  • There’s quite a lot of testing/unstable users, and their proportion is quite stable:
    • Today: testing/unstable: 43119 submissions (31.4%); squeeze: 75454 (54.9%); lenny: 13603 (9.9%)
    • 2011-02-04 (just before the squeeze release): testing/unstable: 29012 (29.7%); lenny: 57262 (58.6%); etch: 10032 (10.3%)
    • 2009-02-13 (just before the lenny release): testing/unstable: 30108 (36.9%); etch: 49996 (61.2%)
  • I don’t understand why the number of Debian stable installations does not increase, except when a new release is made. It’s as if people installed Debian, upgraded directly to testing, and switched back to tracking stable after the release. Or maybe people don’t update their systems? A more detailed analysis could be done by looking at the raw popcon data.
  • Upgrading to the next stable release takes time. Looking at the proportion of users still using oldstable one year after the release, it would be better not to remove oldstable from mirrors too early.

Scripts are available on git.debian.org.