Yes, yes, and more yes. This is all too familiar. I've had the misfortune of having to use this abomination. Sadly, I'm going to have to do so again in the near future, too :-(

http://i.imgur.com/MpUyV.jpg

I have two tables. One represents individual customers. The other represents groups of customers. For the application I'm writing, I want to be able to apply restrictions to certain things, and I want the flexibility to be able to do that at either the customer level or the group level. So each table has a cutoff column, which says that any order with an ID lower than the cutoff should be ignored for a particular operation. So I can quite happily do:

select c.cutoff, g.cutoff from customers c, groups g where c.group_id = g.id and c.id = 45;

However, what I want to do is:

select mymax(c.cutoff, g.cutoff) from customers c, groups g where c.group_id = g.id and c.id = 45;

The problem is, mymax() doesn't seem to exist. I can't use the regular SQL max(), because that's an aggregate function that works across multiple rows. I want to return the larger of two columns in a single row. Oh, and it should know that any given integer is greater than null, because the common case is for there to be no cutoff, and only very rarely will both a customer and a group have a cutoff. I came up with a hideously complicated way of doing it using coalesce(), but it was so nasty I resorted to doing the comparison in python instead. Am I missing something? Surely this should be easy in SQL. I'm using PostgreSQL 8.3, FWIW.

I'm somewhat nervous right now. You see, my main server is moving datacentre. My hosting provider rents rack space, and then resells it to small operations like me. But the datacentre have just doubled their costs, so my hosting provider is taking their business elsewhere. I can't say I blame them. But it means my machine has to move. The problem is, that's something that's fraught with danger. Servers tend to run continuously. On the rare occasions when they're powered down, there's a reasonable likelihood that the drives will fail to work again. Empirically, the chance is far, far higher than for a regular desktop machine that is switched off every so often. The last time I did this, I had 6 drives fail in a 14 drive array. From personal experience, I'd say each drive has only an 80-85% chance of survival.

Of course, I'm a firm believer in not putting all of your eggs in one basket. So my machines are hosted in two separate datacentres, and although the main server is powered off (and I've decomissioned the other one that was in the same datacentre), I still have fully working DNS, SMTP and the critical web sites are all up. In the event that the main server has problems, I can move the non-critical web sites over, too. I have multiple copies of all the data. But still, it's an anxious time. I'll find out if all's OK in about 6 hours. Like I say, fingers crossed...

I have no phone. Its battery ballooned. Furthermore, the spare battery I had is refusing to charge, so I'm currently without a mobile phone. But the thing is, it's surprisingly liberating. I have long despised the telephone. It's an instrument of the devil. This is mostly because I'm not great at holding a conversation. I struggle to think of things to say. I'm really enjoying not being contactable at will. I feels as though a weight has been lifted from my shoulders (one that I didn't realise was there in the first place). Carphone Warehouse not only don't have a new battery for me, but they can't even order one[1]. I've ordered one from a mail order place on the net, and it will no doubt arrive in a few days. But even when it arrives, I'm considering using my phone less. I suspect I'll still take it with me when I'm out and about. But I may well have it turned off much of the time. I like the feeling of freedom.

[1] What's that about? It's not as if it's a prehistoric phone. Sometimes I despair at the increasingly disposable nature of the consumer electronics market.

http://lwn.net/Articles/354860/

Sometimes you can really appreciate living in a world that isn't 100% driven by a company's bottom line. Can you imagine the uproar if Gates, Ballmer or Jobs had come out with a comment like that? Yaaay Linus.

Some while back, spam became a sufficiently large problem for me that I decided to do something about it. After looking around at the options, I decided to go with bogofilter. It's small, fast, and written in C. These are all good things. It looks at each message and builds up a database of words and the frequency with which they appear in spam and ham, and uses that to classify each incoming message. So far so good. It seems to work very well. When it's not sure, it says so, and files the message in an "unsure" folder, and asks me to manually classify those messages. It also uses that clarification to further learn the difference between ham and spam for future messages. This all sounds ideal.

The problem is that my main server was starting to spend a significant proportion of its time in disk wait state, while the bogofilter processes tried to access the word database. The load average hit 100 last night, which was an indication that things had gone badly wrong and I needed to do something about it. I don't know how Berkeley DB is implemented internally, but my guess would be that it's vague hash table shaped, and so the size of the database shouldn't significantly affect lookup times. But I don't know that for sure, and my database had grown to a non-trivial 1.5GB in size. Furthermore, bogoutil was failing to read the database file, possibly indicating that it was corrupt, which might have been causing each bogofilter pass to take much longer than necessary, and hence increase contention for the database file on disk.

So at this point, I decided to scrap the word database and start again. I have a corpus of both ham and spam that I used for the initial training, which I knew wouldn't be perfect, but it was a good start, and I could further train it up over time. Then I left it and went to bed. This morning, I realised just how much work bogofilter had been doing on my behalf. You see, in the 12 hours since I switched over to the new word database, I received 18,000 spam messages that bogofilter was now no longer filtering out for me. I knew the spam problem was bad, but that's at least an order of magnitude worse than I expected.

The next step is to train up bogofilter again so that it filters the majority of those out for me, and given the volume of mail that I'm getting, I think I'm going to need to use bogofilter to feed a Jef Poskanzer style dynamically updated block list that I can pass to blackmilter. 18,000 messages (plus however many bogofilter blocked) in 12 hours? That's insane.

Bizarre, I posted this earlier today, yet it seems to have disappeared. Maybe I just didn't hit submit after the preview. Anyway:

<?xml version=\"1.0\" encoding=\"UTF-8\"?>

Grrr. If you're going to stick that at the top of your feed, please mean it. 0xA0 is Latin1, not UTF-8. Lying to me like that just makes my life miserable :-(

I was looking forward to installing the recently released Fedora 11. But sadly, it was not to be. The installer fails. And by that, I mean it doesn't even get as far as the installer itself. I can't get beyond ISOLINUX. The "Press Enter to begin the installation" message appears to mean "Press Enter repeatedly and bugger all will happen". Sadly, I can't even report it as a bug. The Red Hat bugzilla has decided that my password has expired, but doesn't want to send me a password reset token :-(

I'd go back, find whoever invented multithreading, and blow their brains out before they had a chance to do so. Great when it works, a pain in the arse to debug when it doesn't :-(

Don't get me wrong, SQL is a hideous language that makes it hard to express common things, and impossible to express some more complex things. But today I learned about the coalesce() function, which makes it just that little bit more bearable.

So it came to pass that I ran out of disk space in my server. No problem. Storage is cheap, so I went and bought myself a couple of 750GB disks, from two different manufacturers, to turn into a RAID1 mirror. So far, so good. On a whim, I shut down my desktop machine (which uses the server for NFS/NIS/DNS etc) before taking the server down. Once the disks were installed, the server came back up fine... but my desktop refused to boot :-(

The machine powered up, fans were whirring away, but all I got was a blank screen, not even getting as far as showing the BIOS POST information. No diagnostic beeps. Nothing. A bit of digging and judicious unplugging of everything in turn revealed it to be a bad memory stick. Of course, as is the way of PC hardware, not only does it fail frequently, but when it does, it's considered obsolete, so you can't easily replace it anyway. My local PC components shop no longer sells DDR400 RAM, so I'll have to get some mail order, which will no doubt take forever. In the mean time, I'm limping along with 512MB RAM. Sigh...

A bit late, but who cares?

In the studio

1. Mötley Crüe -- "Saints of Los Angeles"
2. Blaze Bayley -- "The man who would not die"
3. Hanging Doll -- "Reason and madness"
4. Liquid Sky -- "Identity"
5. Guns N' Roses -- "Chinese democracy"
6. Judas Priest -- "Nostradamus
7. Into The Woods -- "Into the woods"
8. Battlelore -- "The last alliance"
9. AC/DC -- "Black ice"
10. Firewind -- "The premonition"

Not a great year, but not a bad one either, and a few surprises. I wasn't expecting Mötley Crüe to come out with such a strong album ever again. Nor was I expecting the GNR album to be any good. Indeed, I wasn't even planning on getting it at all, but it was cheap in the Woolworth's closing down sale. While it's not as good as they were 20 years ago, it's not bad. Of course, there's every reason to call the band Axl And The Hired Guns rather than GNR, but that's an argument for another day. Judas Priest had made a great return to form with their reunion album, so it was a bit of a disappointment to see them follow it up with "Nostradamus". There's nothing wrong with the album, but there's not much memorable about it either. In truth, only perhaps the top four albums in the list are really worthy of their place, and while I like the others, they're nothing special.

There are a few albums from 2008 that I haven't heard yet, but had I done so, I might reasonably expect them to make a showing in my list of favourite albums of the year:

• Sabaton -- "The art of war"
• Falconer -- "Among beggars and thieves"
• Avantasia -- "The scarecrow"
• Alice Cooper -- "Along came a spider"

On stage

For me, 2008 saw 151 bands playing at 34 shows. The highlights were:

1. Rage, The Underworld
2. Sabaton, The Purple Turtle
3. Pain, The Astoria
4. Heaven's Basement, Bloodstock Open Air
5. Blaze Bayley, The Underworld
6. Dio, The Astoria
7. Hanging Doll, The Underworld
8. Liquid Sky, The Peel
9. Metalloid, Bloodstock Open Air
10. Scorpions, Hammersmith Apollo
11. Hellfighter, Bloodstock Open Air
12. Liquid Sky, Barfly
13. Mantra, Bloodstock Open Air
14. Ensiferum, Paganfest, Koko

Another strong year in terms of live performances. I hadn't been impressed with Rage's "Speak of the dead" album, so despite some good performances in the past, my expectations weren't high. Fortunately, they were touring in support of a vastly better new album, "Carved in stone", and on the strength of their excellent performance at The Underworld, I picked up that album at the show. A great choice of setlist, and they even brought out Jen Majura for a few songs, including "Dies Irae", a song I never thought I'd hear them play live.

Sabaton blew me away the first time I saw them, and since then I've been to all of their London shows. Once again, they were outstanding. Pain were something of a surprise. I'd been vaguely aware of their earlier material, but it hadn't really made an impression. But what a live band! They completely blew headliners Nightwish off the stage. Other notables this year were the large number of good performances at Bloodstock, particularly from bands I'd never heard of before (Metalloid, Hellfighter and Mantra stand out there).

Also of note was Liquid Sky's change of vocalist. I thought they'd struggle to replace Hecate, who put in a great performance at Barfly early in the year. Her replacement, Tiffeny, looked competent at Bloodstock, but a bit shaky. Then again, it was her first ever show with the band, and when I saw them again at The Peel later in the year, they were much improved. It was the last date of their tour, and it showed. Having had a bit of practice, she was much more at ease, and far more confident on stage, which made for a great show.

On the screen

This is where I struggle a bit. I didn't see many films this year. Indeed, off the top of my head, I can only recall two. The first was "The dark knight", which was hugely overrated. A decent performance from Ledger, but a poor film overall, and very disappointing after the promise shown by "Batman begins". The second was "Burn after reading". Holy crap. I want those two hours of my life back. How can such an all star cast end up producing such a terrible film?

Since we don't have TV reception, I can't comment on anything there. However, we've been watching a bunch of stuff on DVD. "Heroes" was excellent, and I can't wait to see the second season. I bought the complete set of "The X-Files", which is nearly a week of continuous 24x7 viewing. We've done the first series so far. Also seeing time on the DVD player have been "Magnum", "Dexter", "Doctor Who" and "Torchwood", all of which were good.

Bah. The hard drive on my firewall failed last night. I reinstalled onto a new drive this morning, but I can't get NATing to work :-( Packets are coming in on the internal interface, and leaving fine on the external interface, but aren't being NATed on the way through. I'm using a simplified pf.conf which looks like:

int = "fxp0"
ext = "rl0"

nat on rl0 from 192.168.8.4 to any -> 10.0.8.230

block in log
pass in log quick on $int
pass out log on $ext keep state

But packets are still leaving rl0 with a source IP of 192.168.8.4. Does anyone have any ideas about where I might be going wrong? Jorg?