Those support tasks don't exactly push hardware to its limit, and most of those tasks are the kind of thing that demands a bunch of redundant servers anyway.

Throw a bunch of "last generation" hardware at the task -- stuff from the "asset reclamation" pile. Leave a few more around as spares. Less disposal paperwork. Works just fine. By the time your last spare fails, you'll have a new generation of obsolete hardware.

I've done a little bit of environment taming in my day.

Everybody's already told you the "right" things to do. They're all right. Thing is, you need to get there somehow, and you're looking for a path from here to there. At least, I think that's what you're asking.

You already have bazaar. Good tool. Don't worry about bzr versus cvs versus hg right now. You picked something. Run with it.

I suggest a quick shell script that replaces your editor with "edit; check-in; offer to push". Create another quick script (call it "oops") that asks you whether you need a local or global revert, then issues the relevant commands. Push those scripts to all machines (maybe as their own bzr project). Now, you basically have the same process with a much larger safety net. This isn't software that's being released to the world. Don't worry about version numbers or branching. If you EVER have to change a file, throw it into version control.

Now, you can start to synchronize your scripts. With an environment that wild, you probably have a script that's almost-but-not-quite the same running on a bunch of machines. Maybe there's a hard-coded hostname or directory or something. Come up with a version that's more universal (Use big "if hostname = foo" blocks if you have to), and get that new universal script added to a project and pushed to all machines. Once they're all using it, you can slowly clean it up.

Cool. You have unified scripts. Now let's talk about those configuration files. I'll bet that they're also 99% identical across all of your machines. Get them all into the same project (call them config-machine1, config-machine2, etc.). Get them as identical as possible. Now, think about how you might handle differences. For a quick fix, I like the "magic comment" ("## BEGIN foo.sh MANAGED SECTION" and "## END foo.sh SECTION") and a perl script that looks for those strings. m4 also works well, and isn't too hard to learn.

"Okay, smart guy. I have all of the common config scripts, but I have a bunch of single-purpose machines and scripts, too!" Yup. Awesome. Get them into version control, too. You never know when that machine's going to suddenly die or your boss will break out in a fit of generousity and get you that second server for load balancing. (Hey! It could happen.) When it does, setup will be a lot easier if you have all of the config files in a project.

At this point in the game, you'll be pretty comfortable with version control. You'll have been burned once or twice, and it'll have saved your butt a few times. You'll have some experience, and you'll be kicking yourself for the way that you first set it up. Now's the time to revisit those decisions. Is it time to split up some projects or roll some together? Maybe git or hg might make more sense. Maybe you hate your life and your coworkers so much that you want to go to Perforce, ClearCase, or some other commercial software. You'll have the experience to design it right.

A lot of my biggest concerns have been addressed by others. A few things that I haven't seen covered:

The "30 second boot time" limit makes me assume that there is something time-sensitive about this data collection. (Otherwise, why would you be wasting time on it?) So, you need a fast boot, but then you're mucking around with Samba and union mounts, which are both relatively slow. This doesn't make any sense. This is why people are asking questions or making up odd scenarios in their answers.

The odd scenario that I'm assuming is that you have more drives than sleds, so you need to go through a few load-boot-read-shutdown-unload cycles to get all of your data OR the machine's being "borrowed" to read the data, so you need to bring it up with an alternate OS quickly so that you can work through the night before returning it to normal use in the morning.

If that's the case, it really sounds like (as someone else suggested) that you need to separate the collector from the persistent storage. Set up something that can read the data from all of your "dynamic" drives as fast as possible. Depending on the data, something like rsync or even netcat might be the fastest way to get data off of the machine.

In the US, there are two major "flavors" of cellphone technology: GSM (also used in Europe) and CDMA (also used in some parts of Asia). There are four major carriers: Verizon, AT&T, Sprint, and T-Mobile. There are also some (very good) regional carriers and a whole bunch of companies who re-sell from the big 4.

CDMA providers (in the US, Verizon and Sprint are the two big ones) don't have removable SIM cards, and they're not particularly friendly or helpful about unlocking existing phones. Verizon has, by far, the best coverage for the more rural parts of the US. So, if you're going to be out in the sticks, you might be stuck buying a shitty Verizon phone.

GSM providers WILL sell you a SIM, just like you're looking for, but they don't talk about it, and they don't offer any sort of commission to their store employees for it, so they won't offer it unless you ask. You walk in and buy a pre-paid SIM card, just like in Europe. The data prices suck if you don't get a data package (AT&T charges $2/MEGAbyte without a plan (or $2048/GB), but you can get $2/day unlimited data or pay $25 for a 1GB block of data), but you can get "Unlimited daily" plans for a few dollars per day. AT&T has better coverage than T-Mobile, but both are usually adequate in bigger cities. Neither's quite as good as Verizon in rural areas, but I've had better luck with AT&T than with T-Mobile.

Assuming that you're talking about these: http://www.digistor.com/Blu-ray-Recordable-Media/100GB-BD-R-Printable

That's $1175 (at $47/disk) to back up 2.5TB of data. Or, about $4000 and a whole lot of free time to back up the photos he already has, plus $20 per shoot.

It's a cool option, but not exactly an inexpensive one.

I'd like to see a law that stipulates that any store that offers products in plastic clamshell packaging MUST be willing to open all of the packages in the checkout line (no "go wait in a separate customer service line after paying") at no extra charge. Those packages would be gone within a year.

Right now, clamshell packaging is a huge win for the store, but all of the customer frustration is an externality. By forcing the stores to deal with the externality, we align store interests with consumer interests.

Whether that's true depends, to a great degree, on the environment and the threats that you're defending against.

I work in a secure, guarded building and have to swipe a card just to get to my desk. The odds that anyone else will EVER see me type a password are small. If I write down all of my passwords on a piece of paper that's kept in a locked desk drawer, the risk to the organization is minimal. There's no harm in forcing me to have an absurdly long password that's changed often, as I don't NEED to remember it.

On the other hand, a front-desk secretary doesn't have a private space. We need to ensure that his/her password is easy to remember and rarely changed so that the secretary is NEVER tempted to write it down.

(Personally, I use Keyring for PalmOS. You need to have the device and you need to know my keyring password to get anything else.)

First, don't forget physical security. Assume that someone WILL attempt to steal your netbook. Keep it in sight or locked up. Encrypt as much as you can (whole hard drive if at all possible). Make backups, even if that's just "webmail and flickr/picasa", to keep data loss to a minimum.

That said, I'd keep it simple. Get everything for your online banking set up before you go. Take a look at the certificates. Don't worry too much, but just know whether your bank's certificate has the name of your bank or the name of some parent company. Really, you want to know if something changes later.

Seriously consider two browsers: one for "safe" targeted work (checking bank balance, for example) and one for "browsing". Personally, I'd use Firefox for the safe stuff and Opera for everything else. The Opera Turbo http://www.opera.com/browser/turbo/ feature is really nice for slow or flaky connections.

Like most people, I have a small amount of truly irreplaceable content (documents, pictures) and a whole bunch of "it'd be annoying if I lost that" content (music, movies). One of the really convenient things about this split: the truly irreplaceable stuff is not very large. My docs and pictures occupy about 15 GB, and most of that is pictures.

I have an external hard drive where I back up everything at least nightly. This protects me from accidental deletions and a failed hard drive. It doesn't protect against fire or theft, though.

Services like Mozy and Carbonite offer off-site backup for about $5/month (there are many others -- these are the two best known, I think). I could string together something with a spare drive and a friend, but frankly, it would take a year or two before that approach matched the cost of Mozy et al., and frankly, I just don't WANT to worry about this crap. I'll pay the $60/year to make it someone else's problem.

One interesting option: Crash Plan at http://www4.crashplan.com/consumer/index.html . They offer free backups to friends' machines, and paid backups to their own fileservers. Sounds like the best of both worlds, but I haven't gotten around to trying it yet.