Yes, yes, and more yes. This is all too familiar. I've had the misfortune of having to use this abomination. Sadly, I'm going to have to do so again in the near future, too :-(

http://i.imgur.com/MpUyV.jpg

I have two tables. One represents individual customers. The other represents groups of customers. For the application I'm writing, I want to be able to apply restrictions to certain things, and I want the flexibility to be able to do that at either the customer level or the group level. So each table has a cutoff column, which says that any order with an ID lower than the cutoff should be ignored for a particular operation. So I can quite happily do:

select c.cutoff, g.cutoff from customers c, groups g where c.group_id = g.id and c.id = 45;

However, what I want to do is:

select mymax(c.cutoff, g.cutoff) from customers c, groups g where c.group_id = g.id and c.id = 45;

The problem is, mymax() doesn't seem to exist. I can't use the regular SQL max(), because that's an aggregate function that works across multiple rows. I want to return the larger of two columns in a single row. Oh, and it should know that any given integer is greater than null, because the common case is for there to be no cutoff, and only very rarely will both a customer and a group have a cutoff. I came up with a hideously complicated way of doing it using coalesce(), but it was so nasty I resorted to doing the comparison in python instead. Am I missing something? Surely this should be easy in SQL. I'm using PostgreSQL 8.3, FWIW.

I'm somewhat nervous right now. You see, my main server is moving datacentre. My hosting provider rents rack space, and then resells it to small operations like me. But the datacentre have just doubled their costs, so my hosting provider is taking their business elsewhere. I can't say I blame them. But it means my machine has to move. The problem is, that's something that's fraught with danger. Servers tend to run continuously. On the rare occasions when they're powered down, there's a reasonable likelihood that the drives will fail to work again. Empirically, the chance is far, far higher than for a regular desktop machine that is switched off every so often. The last time I did this, I had 6 drives fail in a 14 drive array. From personal experience, I'd say each drive has only an 80-85% chance of survival.

Of course, I'm a firm believer in not putting all of your eggs in one basket. So my machines are hosted in two separate datacentres, and although the main server is powered off (and I've decomissioned the other one that was in the same datacentre), I still have fully working DNS, SMTP and the critical web sites are all up. In the event that the main server has problems, I can move the non-critical web sites over, too. I have multiple copies of all the data. But still, it's an anxious time. I'll find out if all's OK in about 6 hours. Like I say, fingers crossed...

I don't actually have anything to say. Kathleen is due any day, and I'm looking forward to a few weeks of staying home, getting poor sleep, and changing diapers.
But mostly I'm testing to see if journal saving works properly.

I have no phone. Its battery ballooned. Furthermore, the spare battery I had is refusing to charge, so I'm currently without a mobile phone. But the thing is, it's surprisingly liberating. I have long despised the telephone. It's an instrument of the devil. This is mostly because I'm not great at holding a conversation. I struggle to think of things to say. I'm really enjoying not being contactable at will. I feels as though a weight has been lifted from my shoulders (one that I didn't realise was there in the first place). Carphone Warehouse not only don't have a new battery for me, but they can't even order one[1]. I've ordered one from a mail order place on the net, and it will no doubt arrive in a few days. But even when it arrives, I'm considering using my phone less. I suspect I'll still take it with me when I'm out and about. But I may well have it turned off much of the time. I like the feeling of freedom.

[1] What's that about? It's not as if it's a prehistoric phone. Sometimes I despair at the increasingly disposable nature of the consumer electronics market.

http://lwn.net/Articles/354860/

Sometimes you can really appreciate living in a world that isn't 100% driven by a company's bottom line. Can you imagine the uproar if Gates, Ballmer or Jobs had come out with a comment like that? Yaaay Linus.

We've made some significant updates to the submission/journal system. Visiting Submissions and Journals yields a new form that allows stuff like tags to the data types. There are a number of annoying bugs, but for the most part the dust is starting to settle. More notes will be coming, but this journal entry is really just me putting the final test on the new Journal form.

I saw recently that FreeBSD 8 was in BETA state. I ran 7-CURRENT for a while, because it had features I wanted to test (improvements to the OSS implementation mainly), so I thought I'd give it a try.

This time, rather than doing my usual source install, I tried a binary upgrade using freebsd-update. What a disaster. While the source upgrade procedure uses mergemaster to update configuration files, letting you just keep the new version of files you haven't modified, freebsd-update makes you merge them all by hand where there is a conflict. This wouldn't be a problem, except that all of the config files have a version line at the top, which conflicts between the two versions.

Inevitably, when manually handling the merge for a few dozen files, I missed an important bit so my first boot failed with an error complaining about the diff lines still being in the file. I fixed that, and rebooted.

My next boot failed because one of the startup scripts had replaced an if statement with a case. Unfortunately, this hadn't shown up as a conflict, so it had just taken the start of the case statement and the end of the if, giving nonsense. Fortunately, I was able to find the correct version in CVS and copy it out.

Next boot, my network interfaces weren't working. Actually, this was a problem I'd found earlier. When you update FreeBSD, you update the kernel, reboot, then update the userland (the new kernel is guaranteed to support the old userland, but the converse is not true). The em driver for Intel GigE cards complained that they both had invalid MAC addresses. Not a huge problem; it's a VM so I could just change the kind of virtual network card it was providing to the machine, but checking the bugs database I discovered that it's giving the same error for people with ThinkPads that actually do have this kind of hardware built in. Great.

Finally, my system decided to fail to boot with the error:

mounting /etc/fstab failed, startup aborted

Strange, I thought, I wonder which disk is failing to mount. A quick check in single-user mode showed that everything in fstab had mounted correctly. I eventually tracked this down to a bug in /etc/rc.d/mountcritlocal. This is not present in CVS, so it's probably introduced by the merge process. The value of $? (the exit value from the last command) is stored in $err, another command is run, and then there is supposed to be a switch statement branching on $err, which instead is branching on $?.

I've run betas, release candidates, and even the development branch of FreeBSD before, but 8-BETA2 is the first time I've ever had a FreeBSD install that feels like a beta. The merging done by freebsd-update seems completely broken; it prompted me for things it could have trivially done automatically, but failed to prompt me when it broke random system files. My system is now working again, but it's irritating to have to spend this much effort on an update.

Some while back, spam became a sufficiently large problem for me that I decided to do something about it. After looking around at the options, I decided to go with bogofilter. It's small, fast, and written in C. These are all good things. It looks at each message and builds up a database of words and the frequency with which they appear in spam and ham, and uses that to classify each incoming message. So far so good. It seems to work very well. When it's not sure, it says so, and files the message in an "unsure" folder, and asks me to manually classify those messages. It also uses that clarification to further learn the difference between ham and spam for future messages. This all sounds ideal.

The problem is that my main server was starting to spend a significant proportion of its time in disk wait state, while the bogofilter processes tried to access the word database. The load average hit 100 last night, which was an indication that things had gone badly wrong and I needed to do something about it. I don't know how Berkeley DB is implemented internally, but my guess would be that it's vague hash table shaped, and so the size of the database shouldn't significantly affect lookup times. But I don't know that for sure, and my database had grown to a non-trivial 1.5GB in size. Furthermore, bogoutil was failing to read the database file, possibly indicating that it was corrupt, which might have been causing each bogofilter pass to take much longer than necessary, and hence increase contention for the database file on disk.

So at this point, I decided to scrap the word database and start again. I have a corpus of both ham and spam that I used for the initial training, which I knew wouldn't be perfect, but it was a good start, and I could further train it up over time. Then I left it and went to bed. This morning, I realised just how much work bogofilter had been doing on my behalf. You see, in the 12 hours since I switched over to the new word database, I received 18,000 spam messages that bogofilter was now no longer filtering out for me. I knew the spam problem was bad, but that's at least an order of magnitude worse than I expected.

The next step is to train up bogofilter again so that it filters the majority of those out for me, and given the volume of mail that I'm getting, I think I'm going to need to use bogofilter to feed a Jef Poskanzer style dynamically updated block list that I can pass to blackmilter. 18,000 messages (plus however many bogofilter blocked) in 12 hours? That's insane.

Does anyone remember Frontier, a space trading game from the '90s? No, not that one, but a much lesser-known top-down game that only ran on Windows NT. It was released back in '95 and I found it a couple of years later when I was running NT 4 on my PC.

The game was incredibly addictive, but it was unfinished. The version I had was 0.5, and Altavista (this was a few years before Google) was unable to find a newer version. The gameplay owed a lot to games like Nethack. You started off in one solar system and then got to the next through a jump gate (analogous to descending to the next dungeon level). Over time, you'd upgrade your ship, with better shields and weapons, and progress further. Being a 0.5 release, there were a few things missing. The lack of sound was a shame, but the real killer was that there was no save system. You could play for an hour, then get hit by a stray nuclear warhead and have to start from the beginning. A game with so much potential, but it never went anywhere...

...or so I though. Over the weekend, some random googling turned up the author's web site and it turns out that he has recycled a lot of the ideas into a brand new game: Transcendence. This has a improved graphics, sound, and working savegames (nicely integrated into the game so they aren't a crutch). The story line is much expanded on Frontier (which was basically 'you are in space. Have fun') and the universe is much richer. Things I liked in the original, like the randomly-generated solar systems, the black market and the different possible gameplay styles are all still there, but now there is a rich backdrop and the player can choose to help the military, fight pirates, provide comet-grown food for expensive restaurants, or any combination.

There's one down side: It's still Windows-only, and I don't have a Windows machine anymore. Fortunately, it runs very well in WINE. I've playing it on the Mac in the free version of CrossOver Games that was released last year.

Oh, and if anyone's interested, you can still download Frontier 0.5. It does have one advantage over the newer game; the AI didn't have any sensible friendly-fire logic, so you could easily destroy (and loot) friendly space stations by getting one of the ships defending it to fire while docked. This was easy to do: just get the pirates to chase you there and when their stray shots hit the station all of the docked ships will launch firing. This works really well for the black market outpost, which is protected by very powerful ships and is full of fun technology to steal.

Bizarre, I posted this earlier today, yet it seems to have disappeared. Maybe I just didn't hit submit after the preview. Anyway:

<?xml version=\"1.0\" encoding=\"UTF-8\"?>

Grrr. If you're going to stick that at the top of your feed, please mean it. 0xA0 is Latin1, not UTF-8. Lying to me like that just makes my life miserable :-(

I was looking forward to installing the recently released Fedora 11. But sadly, it was not to be. The installer fails. And by that, I mean it doesn't even get as far as the installer itself. I can't get beyond ISOLINUX. The "Press Enter to begin the installation" message appears to mean "Press Enter repeatedly and bugger all will happen". Sadly, I can't even report it as a bug. The Red Hat bugzilla has decided that my password has expired, but doesn't want to send me a password reset token :-(

I'd go back, find whoever invented multithreading, and blow their brains out before they had a chance to do so. Great when it works, a pain in the arse to debug when it doesn't :-(

Don't get me wrong, SQL is a hideous language that makes it hard to express common things, and impossible to express some more complex things. But today I learned about the coalesce() function, which makes it just that little bit more bearable.