I ended up making some tiny changes to my WP install that basically causes requests to /wp-admin to die immediately, unless you're accessing it via a specific HTTP port that I've opened in Apache specifically for this purpose.
I've got disk permissions set up so that the regular Apache user cannot write at all to the disk - a common source of WP problems seems to be exploits writing new files to disk, so stopping that seemed like a good idea. Unfortunately it also bones a lot of WP functionality like being able to automatically install skins/plugins.
Using some Apache module (can't remember which one) I've set it up so that requests made to /wp-admin under the correct Apache port operate under a different user - one that /does/ have write access to the disk. So it means I can do any administrative stuff and take advantage of the full WP functionality without having to leave write access in there for normal use.
Conceptually this seems like a much more default setup for WP - certainly I haven't had any security problems. As a side benefit it means I don't need to worry about random attacks like this.
There's a few minor problems I haven't resolved (most notably when adding new posts, the URL it stores for them includes the administrative port in them and publicly displays them in things like the RSS feed :) but I'm hoping to find time one day to resolve those.