Last month, my work got a new H.323 video conferencing unit, and today we had our first real test: a lecture given at SFU that was streamed to us. For the most part, it went really well; there were no big screw-ups and everything went as planned. During the second half of the conference, though, the audio was intermittently choppy. I'm not certain, but I think that a local user's Internet radio stream may have caused the problems.

If that's the case and it would surprise me, since I'd assumed we had a pretty damned fast connection to the Internet then I'll need to start adding traffic shaping to our firewall. Working on the firewall is something I've been putting off for a while, since it's a bit obscurelovely pf firewall, littered through with quick rules. But there's a good tool for pf unit testing I've been meaning to try out since I heard about it at LISA. Probably won't be as big a help with the traffic shaping stuff, but at least I'll be reasonably sure I'm not screwing anything else up.

And now I'm wondering just how hard it would be to come up with (handwave) something that would combine automatic form generation, web-based testing code and summary code. We have these multiple conferences that need registration pages; while some of the information is the same (name, email address) some is different (one conference has a banquet, another wants to know if you're going to be attending all three days). Putting all this in a database and using something like Formitable to generate the form seems perfect.

Since I'm already using Perl's WWW::Mechanize and Test::More to test the pages, it'd be nice to have it look at the stuff used to generate the form and use that to test the page. (That's not the clearest way I could put that, but if I don't write this down now I'll never write it down.) And if I could add something that'd automatically generate summary pages for conference organizers, it'd be even better; stuff like email and address is always easy, but being aware of special questions would be nice too. (Though maybe not necessaryhow hard is it to generate summary pages?)

Trouble is, this is a lot of deep thinking that I've never really had to do before. I suspect this sort of thing is a good programmer's bread and butter, but I've never been a programmer (good or otherwise). The more I think about this, the more I can't decide whether this is really hard, possible but too much effort to be worth it, or already done by something I haven't come across yet.

The little things I can handle, though. This crash looks like it's happening because of a mixup between rand(3) and random(3). In Linux, both have a maximum of RAND_MAX, but in Solaris the latter has a maximum of 2^31. This wreaks havoc with the let's-shuffle-the-playlist routine in XMMS, and we end up with a crash. Once I figure out how to program in C, it shouldn't be too hard to get it fixed. :-)

This was Arlo's first time going all the way under water. He was definitely surprised by the whole operation, but he didn't panic or cry or anything. Such a trooper.

I installed RT at work a couple days ago using pkgsrc. This was the first time I'd ever used pkgsrc, and I have to say I'm impressed. Yes, it's just like a portable ports tree -- but it's just like a portable ports tree, and I'm starting to think that's a very, very powerful idea.

RT went well except for the final install, where it complained and died. Fortunately, it turned out to be susceptible to exactly the sort of one-line patch that I have an affinity for. Not as cool as correcting Theo de Raadt's code, mind you :-) but still a good feeling.

Ah...RT, I've missed you.

• Word-by-word diff of GPLv2 and v3
• New version of Unix Backup and Recovery, by W. Curtis Preston. The first was amazing; this one covers Windows and Mac OS X as well. WANT.
• Good reading and good suggestions. I need to alias ls, fg, mutt, sudo, and to shorten my alias for SSHing to my webserver.
• I think I finally figured out what was going on when my desktop machine at work had a suddenly borked mouse/X relationship: double-clicking was inconsistent, single-clicking was inconsistent, but the keyboard still worked. It happened again a few days ago, and I managed to run xev and verify that things really were messed up: the mouse and all its clicks and motions just weren't registering at all. I managed to figure out what was (probably) going on when I came across this link. In the end, what worked for me was switching to a console and running cat /dev/input/mouse1. Don't know why (yet!) but that seemed to reset everything.

Still to come: Why upgrading is the most important thing EVAR.

People have been calling me out on my last post, and that's good; I love a good argument^Wdebate, and doubly so when it comes from people w/more experience than me. So I'm going to start responding to the comments, laying out where I'm wrong and where I still think I'm right.

I said:

OpenSolaris: If I wanted to upgrade everything by hand, I'd stick with Slackware.

Bzzt! As I found on on a recent episode of BSDTalk, NetBSD's pkgsrc is available for over nine hundred thousand operating systems, including Solaris and Slackware Linux. Tha's right, both premises in that statement were wrong.

Not only that, pkgsrc can be tucked out of the way so that it doesn't interfere with the rest of the system...so I could even throw it on Thornhill right now, Slackware and all, and start using it instead of my own half-assed build script for Apache/SSH/PHP/OpenSSL/mod_ssl (which, in my own defence, works pretty darned well).

In fact, tomorrow I'm heading out to The Other University to set up two new X4200 servers, and I'm seriously considering adding pkgsrc to them -- if only to avoid having to compile (and botch) Lapack and Blas. If that goes well, I may start adding it to the main server here so that we can easily get more up-to-date versions of Firefox et al. (Though I could probably get them from Blastwave...this has been a low enough priority for me so far that I haven't really looked into all my options.)

That is not to say it's perfect:

• While it has six thousand packages or so in its tree, only (...) something like two or three thousand compile.
• Upgrading is less than perfect; as you'd expect, the process is basically remove-and-recompile...and since that goes for dependencies as well (at least in the default case), it can potentially be a while before it all gets back to a useable state:

  It is possible, and in the case of updating a package with hundreds of dependencies, arguably even likely that the process will fail at some point. One can fix problems and resume the update by typing make update in the original directory, but the system can have unusuable packages for a prolonged period of time. Thus, many people find 'make update' too dangerous, particularly for something like glib on a system using gnome. To use binary packages if available with "make update", use "UPDATE_TARGET=bin-install". If package tarball is not available in ${PACKAGES} locally or at URLs (defined with BINPKG_SITES), it will build a package from source. To enable manual rollback one can keep binary packages. One method is to always use 'make package', and to have "DEPENDS_TARGET=package" in /etc/mk.conf. Another is to use pkg_tarup to save packages before starting.

  From the Swedish NetBSD wiki. Though it's nice that manual rollback is doable; that's always my big paranoia when it comes to source-based upgrades.

• Upgrading Gnome in particular is a fucking bear.

That last complaint is not as fair as it could be. I mean, I'm not going to be upgrading Gnome on either Thornhill or the two new Sun machines. And at around 80 packages, it would be damned difficult to try and recompile it all without starting with a clean slate. But this sort of nonsense with Gnome is what put me off the ports tree in FreeBSD.

(I was going to put in something about how Debian doesn't need that sort of thing, but I should research that first.)

First off, a special catch-up entry for my very, very special Slashdot friends. :-) I've been slack at cross-posting entries from my blog since about halfway through LISA. I'm going to do my best to fix that, but I won't be posting old entries here...too much work, and Slashdot has no way of backdating old entries.

I've set up a new blog on my site, using a combination of Perl, ASCIIdoc, Make and email. It's working well, though there are some bugs and missing features. The main things I like about it are that it's plain text and there's no PHP. At last, I can edit things in emacs without using Mozex!

And so, on to today's entry in which I make a fool of myself by denigrating different operating systems based on experiences from three years ago. As always, there are comments both here and on the website...including a FreeBSD committer who cordially invites me to put my money where my mouth is. Fair point, and I'll be responding to that later. (Along with admitting my ignorance about pkg-src on Solaris.)

On with the show!

libpst is a command-line tool that converts Outlook .pst files into standard mbox files, the way T&R intended.Wish I'd known about this before... One of the outstanding feature requests is listing and extracting individual messages. Maybe I'll take a look at this.

In other news, I borked my home machine (Debian testing) by trying to extend a partition w/ReiserFS. That gave me a perfect excuse to upgrade to a bigger disk and reinstall Debian.

Next up is maybe looking at replacing my venerable copy of Slackware 9 with a Debian install, too; the ease of installing and upgrading Debian packages is just too good to pass up.

I did consider other OSs:

• FreeBSD: Even after three years, port-upgrade still scares me.

• NetBSD: meh, what's exciting about that?

• OpenBSD: Secure, yes, and God knows I'd like to use pf. But not easy to upgrade, either ports or releases.

• Dragonfly: Not yet.

• BSDs in general: I want a journalled FS.

• OpenSolaris: If I wanted to upgrade everything by hand, I'd stick with Slackware.

And yes, I realize I'm damned ignorant, and that a server should not be exciting. But I'm convinced that a big part of running a server successfully is ease of upgrading, whether security fixes or new app versions, and Debian is just wonderful.

Here's a fun game: create a large (>1GB) file in your home directory called core and start Firefox. Have a look at this part of run-mozilla.sh:

if [ -x "$crc_prog" ]
then
DEBUG_CORE_FILES=1
fi
if [ "$DEBUG_CORE_FILES" ]
then
crc_old=
if [ -f core ]
then
crc_old=`$crc_prog core | awk '{print $1;}' `
fi
fi
##
## Run the program
##
"$prog" ${1+"$@"}
exitcode=$?
if [ "$DEBUG_CORE_FILES" ]
then
if [ -f core ]
then
crc_new=`$crc_prog core | awk '{print $1;}' `
fi
fi
if [ "$crc_old" != "$crc_new" ]
then
printf "\n\nOh no! %s just dumped a core file.\n\n" $prog
printf "Do you want to debug this ? "
printf "You need a lot of memory for this, so watch out ? [y/n] "

Care to guess what'll happen? That's right: Firefox will take 10 seconds to start up because it's busy md5summing a big-ass core file. The user will think that it hasn't launched at all and will click again. Rinse and repeat, with more and more clicking every time. By the time I figured out what was going wrong, the system load was about 200. Fortunately, it's a simple thing to add DEBUG_CORE_FILES= judiciously (not DEBUG_CORE_FILES=0; I keep forgetting that a simple [ $FOO ] simply tests whether $FOO is empty, not whether it's non-zero).

Also: the advantage to being in a small shop is that if you're the only one running Linux on the desktop, you can just go ahead and add things like the latest version of Firefox (now without the amusing bug that makes a search work on some other random tab, instead of the one you're looking at) and the MySQL DBD connector for Perl. It's really incredible how much irritation those two things are gonna save me.

Finally: this is just plain cool. As he did during the Bash scripting BOF, Wout takes me to school. Didn't know about: ssh -t, COLUMNS/LINES environment variables, tput, or just how much Applescript can do.

Memo to myself: Don't eat the Turkey sashimi.

In other news: I don't usually post links to things just to say "go read this". However, I'll make an exception in these cases.

First, I was recently going to use the word "Manichean" to mean "dualistic, good-vs-evil view of the universe, with an implied inevitable battle between the two". However, when I Googled for it to check the spelling, I came across this article explaining why that wasn't a terribly accurate use of the word. Interesting stuff...I certainly didn't know there were any Buddhist-influenced ascetics hanging around Baghdad in the 3rd century.

Second, there's some interesting and contradictory stuff on the procedures for GPG/PGP keysigning parties here and here. Why does publicizing a public key "slightly reduce the security of a key pair"? I don't know. I've had a quick look through my copy of Applied Cryptography (3rd Ed.), donated by the kind man behind Pangolin Systems, but can't find anything from Saint Bruce about this. Anyone?

Third, there's an excellent set of tools for keysigning parties available here. One of the people who signed my key at LISA had used caff to send it back, which is a nice wrapper around the whole procedure (grab the key, sign the key, encrypt the key with itself, email it back to each of the key's email addresses). The lack of understandable (but see next paragraph's self-ass-kicking) documentation for GPG means that a) this automation is very nice, and b) I'm kicking myself for not buying Michael Lucas' book from the No Starch Press booth at LISA.

Fourth, if'n you've got GPG, it's worth reading the documentation, like the FAQ or the GNU Privacy Handbook. Shame on me for not doing that previously. (And shame on me for taking so long to email people's keys back to them.)

Fifth, you can find some pretty stats here, or the trust path from me to Wietse Venema. Geek Pride!

Sixth and finally, there is this handy little page about how to set up a CPAN library in your home directory. Since it took me a while to track this down, I'm throwing it in here so's I can find it quicker next time.

I've just come across AsciiDoc, and this is SO CLOSE to what I want: Ascii-based markup, still intelligible, and rendered into pretty CSS-compliant whatnot.

For a while now I've been toying with the idea of leaving WordPress behind and just writing all my stuff in Emacs, the way RMS intended, and converting it all to pretty HTML through <handwave>some sort of script or Makefile</handwave>. But this...this is perfect. See this? If it were a black monolith orbiting Jupiter, I would say "My God, it's full of stars!" It's clean, it's spare, it looks good, and it does not require verdammnt patching to stop it from throwing in br tags every time it sees a newline. And you know what it requires? Python! That's it!

I know what you're saying: this is like wiki markup without the wiki. EXACTLY! It's easy to write, easy to read, it looks good and it's just static: no PHP remote inclusion waiting to happen, no heavy load, just simple plain text and html. Oh yes.

One of the great things about going to LISA is that you get the proceedings and/or training for everything on CD or dead tree. (Well, nearly everything...I've heard that some people didn't or couldn't make their training materials available (though I've not been motivated to confirm this yet), and some of the talks didn't do this (Tom, where are your slides?)). There is some wonderful stuff to be found in them...

...like WWW::Mechanize, which is just perfect for testing out this conference registration form I'm working on. Only I've run into a bug that comes when trying to specify which button to click on:

$agent->click_button(value => 'Okay to submit');

That li'l chunk gave me this error:

Can't call method "header" on an undefined value at /home/admin/hugh/perl/lib/perl5/WWW/Mechanize.pm line 2003.

One guy reported the same trouble, but got no response. And the RT queue is fulla spam.

But aha, I found out how to use the Perl debugger in Emacs (M-x perldb. Shhhh!) and was able to track things down. Turns out there are a couple things going on:

1. In the page that I'm parsing, there are actually two forms, not one; one sends you back to correct mistakes, one sends you forward to keep going. Since I was not specifying which one to use, it used the first...and in that one, there is no button labelled "Okay to submit". One I specified the right form ($agent->form_number(2);) everything was good.
2. But of course, this sort of thing shouldn't happen, right? Right.

There are a couple subroutines/methods in this module that aren't testing for the right number of arguments. One of 'em is click_button, which has this loop:

my $request;
.
.
.
elsif ( $args{value} ) {
my $i = 1;
while ( my $input = $form->find_input(undef, 'submit', $i) ) {
if ( $args{value} && ($args{value} eq $input->value) ) {
$request = $input->click( $form, $args{x}, $args{y} );
last;
}
$i++;
} # while
} # $args{value}

return $self->request( $request );

No test/case for not finding a button named whatever, so it just blithely returns $self->request( $request ). But of course, request does the same thing:

sub request {
my $self = shift;
my $request = shift;

$request = $self->_modify_request( $request );

if ( $request->method eq "GET" || $request->method eq "POST" ) {
$self->_push_page_stack();
}

$self->_update_page($request, $self->_make_request( $request, @_ ));
}

Again, no test for the right number of arguments. And having just read the Test::Tutorial manpage, I'm all about unit testing and such, baby.