So, a fairly significant chunk of the north of England has just been without internet - DSL was out, some leased lines were down - for three hours. This was, apparently, caused by a single hardware fault in a single hub in Manchester operated by BT.

The fact that a single fault could take down thousands of net connections is not only mindboggling to me, it's more than a little scary. Where are the failover systems? Where is the network redundancy to ensure traffic is routed around failed nodes, even major ones?

Obviously nowhere, because they're too busy cutting corners.

I'd hope that they'd learn something from this, but it isn't the first time this has happened - a couple of years ago, most of the north was plunged into technological blackness when a fire broke out in a single cable trunk in Manchester, and there was no backup or route around it.

This is, frankly, no way to run a railroad. Or a telecommunications network, either.

By way of vague content, something that may be of use to users of doxygen and git:

Code:

#!/bin/sh

# Get the latest tag annotation out of git.
VERS=`git tag -n1 | tail -n1 | perl -e '$tag = <STDIN>; $tag =~
s/^.*?\s\s+(.*)$/$1/; print $tag;'`

# Generate the documentation with the project number updated with the tag.
(cat Doxyfile; echo "PROJECT_NUMBER = \"$VERS\"") | doxygen -


This little shell script will replace the PROJECT_NUMBER configuration setting passed to Doxygen with the contents of the last annotated tag in the current git tree (it won't work with lightweight tags, but I generally don't use those anyway). It's a pretty handy way to make the title bar in generated documentation display the tag of the tree it was generated from.

I guess I should probably put something here, huh?

Perhaps one of the most important developments since my last post here has been my discovery of uinionfs, and how it saves so much pain when updating mediawiki. Unionfs is a filesystem service for linux (and some other *nix OSen) that allows multiple directories to be layered on top of each other, producing a single coherent filesystem. What this means in practice is that, instead of requiring a physically separate MediaWiki installation for each wiki I manage, I can have a single MediaWiki directory containing the common code (that is, everything but the LocalSettings.php configuration file, and the images directory into which uploads are placed), and two directories for each wiki: one directory contains the wiki-specific data, the other is used as a unionfs mount point. The idea is that, for each wiki, I mount the mediawiki directory read-only as the 'bottom' level, and the data directory on top of it, and then make the web server look at the mount point for the wiki. With the addition of a simple script to go through each wiki and run its maintenance/update.php script, I can upgrade all my wikis in seconds instead of taking 10 or 15 minutes to do them all: apply the update patches to the common mediawiki directory, run the update script to update all of the wikis, and it's done.

I've also released the first public version of my Tardis backup system. The basic idea is that you run the system regularly - every hour or so, for example - and it incrementally backs up your system to somewhere else (intended to be a remote server). Each backup only transfers any changes made since the last backup, but when the backups are mounted and viewed, each snapshot looks like a complete backup from the time the backup was made. The techniques behind it - use of cpio, rsync, and other standard tools - are fairly well established Out There, and there are systems that do similar jobs, but I had some specific requirements that none of them provided, or didn't do in the way I needed (like database dump backups). So, yes; backup tool. It can be looked at in my gitweb install or you can grab a tarball.

It's basically Time Machine, but a bit clunkier and less user-obsequious, hence the name!

I'm also deep in code hell with the Course Processor for work. Some stats for that:

Code:

Scanned 12831 lines (564600 characters) in 27 files
Shortest file: 25 lines (./modules/ProcessorVersion.pm)
Longest file: 2648 lines (./plugins/HTMLOutputHandler.pm)
Average file length: 475 lines (20911 characters)
Number of functions: 320
Av. functions per file: 11
Minimum function length: 1 lines (./modules/Template.pm::set_error)
Maximum function length: 188 lines (./plugins/HTMLOutputHandler.pm::preprocess)
Average function length: 24 lines
Longest line: 214 characters (./modules/Metadata.pm line 175)
Comments: 232298 of 564600 characters (41.14%)
Whitespace: 158302 of 564600 characters (28.04%)

File length histogram
Frequency
.---------------------------------------------------------------
0 | ### (3, 11%)
110 | ##### (5, 18%)
220 | ###### (6, 22%)
330 | ##### (5, 18%)
L 440 | ## (2, 7%)
e 550 | ## (2, 7%)
n 660 | # (1, 3%)
g 770 | (0, 0%)
t 880 | (0, 0%)
h 990 | # (1, 3%)
1100 | (0, 0%)
i 1210 | (0, 0%)
n 1320 | # (1, 3%)
1430 | (0, 0%)
b 1540 | (0, 0%)
y 1650 | (0, 0%)
t 1760 | (0, 0%)
e 1870 | (0, 0%)
s 1980 | (0, 0%)
2090 | (0, 0%)
2200 | (0, 0%)
2310 | (0, 0%)
2420 | (0, 0%)
2530 | (0, 0%)
2640 | # (1, 3%)
`---------------------------------------------------------------

Function length histogram
Frequency
.---------------------------------------------------------------
0 | ################################### (71, 22%)
8 | ################################## (69, 21%)
L 16 | ################################ (64, 20%)
i 24 | #################### (40, 12%)
n 32 | ########### (23, 7%)
e 40 | ###### (13, 4%)
s 48 | ####### (15, 4%)
56 | ## (4, 1%)
p 64 | ## (5, 1%)
e 72 | # (3, 0%)
r 80 | # (2, 0%)
88 | ## (4, 1%)
f 96 | (1, 0%)
u 104 | # (2, 0%)
n 112 | (1, 0%)
c 120 | (0, 0%)
t 128 | (0, 0%)
i 136 | (1, 0%)
o 144 | (0, 0%)
n 152 | (0, 0%)
160 | (1, 0%)
168 | (0, 0%)
176 | (0, 0%)
184 | (1, 0%)
`---------------------------------------------------------------

Line length histogram (empty lines omitted)
Frequency
.-------------------------------------------------------------
0 | ###################################################### (1192, 11%)
9 | ################################# (729, 6%)
18 | ############################################################ (1324, 12%)
27 | ####################################### (873, 8%)
C 36 | #################################### (808, 7%)
h 45 | #################################### (811, 7%)
a 54 | ################################# (733, 7%)
r 63 | #################################### (807, 7%)
s 72 | ############################################# (1007, 9%)
81 | ################################# (735, 7%)
p 90 | ################# (374, 3%)
e 99 | ########## (229, 2%)
r 108 | ######## (181, 1%)
117 | ######## (183, 1%)
l 126 | ##### (123, 1%)
i 135 | ### (79, 0%)
n 144 | ## (59, 0%)
e 153 | ### (76, 0%)
162 | ## (51, 0%)
171 | # (31, 0%)
180 | # (27, 0%)
189 | (14, 0%)
198 | (14, 0%)
207 | (5, 0%)
`-------------------------------------------------------------

Last night, after several days of steady hair loss, I finally managed to release the first version of my library that allows SDL_Surfaces to be saved, either to files or SDL_RWops, as jpeg or png data. This self-imposed torture¹ was needed because, while there are several ways to get images into SDL - SDL_image being the most commonly used and widely recognised library for doing this - support for actually saving from SDL surfaces to files is, at best, poor. SDL itself provides a function to save surfaces as BMP files, but if you want to save in any sane format, you're basically stuck with rolling your own.

The save code I have written is in its early stages: as it stands, it supports writing as png and jpeg, as those are the formats I need right now. I have plans in the works to add TGA, TIFF, and possibly PCX (although the latter is a long, long way down the priorities), but supporting saving to other formats that SDL_image supports² is not really something I have any interest in as they are all of questionable utility in real situations.

I have designed the library such that the code needed to save each format is nearly encapsulated in its own file, and with naming schemes that betray a deeper motive: once I have more formats (TGA and TIFF at least) added, and have tested the code more thoroughly, I will be looking into making a major patch against SDL_image, and contacting the maintainer of that library to try and get my code rolled into it. The lack of save facility has been a deficiency in the library for a long time, and hopefully I can start to address that properly. I can at least try, anyway!

Anyway, the release message can be read here. There you will find links to downloads, git repository links, bug reporting facilities, and more. Now I go to start integrating this into a Real Program...

¹ Seriously, what is it with image library developers and documentation? They appear to be decent enough at writing conversational introductions to the use of the library, but if you want actual API docs, with full explanations of function purposes, arguments, and return values? Aahahah, no.
²CUR, GIF, ICO, ILBM, PNM, XCF, XPM, XV

Over the past week, work has seen me looking long and hard at some of the MediaWiki code¹ in order to add more explicit visibility of group membership (every time a user link appears, it includes information about the groups the user is a member of), and group filtering facilities to the Recent Changes and Contributions pages, and their associated feeds. This entry is an attempt to document the overall changes made, primarily so that I have a reference down the line, but it may be of interest to Like Minded Lunatics.

A side issue here is that the changes I've made are all against the 1.15.4 stable release, so realistically I can't actually submit the patches to the Foundation (despite the fact that, for example, group filtering of Recent Changes has been a feature request since 2006!) Having checked out the current subversion tree, I have found that some significant changes have been made to some of the files I have patched², but the majority will simply involve me copy&pasting my changes into place, and one change even removes the need to patch one file entirely, so I could conceivably insert my changes into the latest code and work out patches from that... I may consider that next week sometime.

But anyway, for now I will leave the whole issue of explicitly displaying group membership, probably for another entry, and will concentrate on the group filtering of Recent Changes and Contributions. My current patch modifies four files to add the filtering: ChangesFeed, SpecialContributions, SpecialRecentchanges, and MessagesEn, the latter being required to add two messages rather than hard-code the text shown in forms.

The first change I had to make was in includes/ChangesFeed.php, the code that is used to generate feeds for Recent Changes and Contributions. In the stable release³ the problem I encountered was that the caching being done to reduce feed generation overhead was too 'dumb': the code attempts to generate a memory cache key based on only three of the potential query string arguments - the number of items to show, whether minor edits should be shown, and the target. Amusingly, it appears that the maintainers of ChangesFeed and SpecialRecentchanges were somewhat opposed to talking to each other, as SpecialRecentchanges labeled the behaviour of ChangesFeed as a bug, when in fact it was merely a side effect of the latter code not accepting the full range of possible recent changes settings when working out its cache key. I needed to expand the arguments it was aware of so that it would correctly generate and cache the correct feed when one of my new arguments was specified. In the end, I did it the lazy way, and just added extra parameters to the appropriate function - the cutting-edge code handles it in a rather nicer fashion by taking an array of options, and I may backport that code.

The changes to includes/special/SpecialRecentchanges.php are more extensive. Filtering on group requires that the code is actually aware of groups! First it needs a 'group' argument added to the recognised query string arguments, and a select box adding to the form through which the user controls the settings to allow specification of a group. The code to add group filtering to the database query actually proved fairly simple - a bit of code to add a condition to the query if a non-implicit group is selected, and another bit that joins in the user_groups table if needed. While working on adding the form changes, I was originally just allowing for filtering on non-implicit groups (ie: groups other than 'user' and 'anonymous'). After running into the feature request linked above, I decided that I needed to add the ability to filter on the user and anonymous groups, but I also realised that I could cheat, as the ability to filter on those two groups is already there: the controls already allow you to hide or show anonymous edits and user edits, all I had to do was detect when a user had selected to filter on 'user' or 'anonymous' and then flip the appropriate flags in the code that generates the database query!

Fairly similar levels of modification were needed in includes/special/SpecialContributions.php - again, the addition of a new query string option was needed, and the supporting form modifications. Unlike SpecialRecentchanges, this class doesn't use the MediaWiki 'FormOptions' class to handle arguments, so some additional work was needed to handle the group filtering option. The database changes actually get pushed down into the ContribsPager class, and mainly consist of adding an extra condition and table join if a group filter is specified.

Overall the changes are fairly simple, something I wasn't expecting when I started it (a good three days of the time I spent on the problem were given over to getting my brain into the same space as the MediaWiki developer's, to work out how to add the functionality). The patch file is 15K unpacked, but actually a lot of that is patch context, the meat of the changes is really much smaller - I was surprised, given how long the issue has remained unaddressed. But now, I need to refactor the bulk of my course processor's output code...

¹ which is the usual mixed-bag of quality I've come to expect from major projects. Some files are quite exemplary, while others are vaguely nightmarish. The worst ones aren't as bad as some code I've seen, but even the best ones make me realise that my habit of making comments take up around 30% of any source is deeply atypical.
² none of the changes actually replicate the functionality my work has added, so it hasn't rendered the overall project redundant, thankfully.
³ but not, as far as I can tell, in the cutting-edge code. It appears that the ChangesFeed in the subversion tree completely overhauls the handling of arguments to some key functions that replicate the functionality I had to add, so it looks like people have actually started talking to each other...