For more good ideas like this, watch this screencast from pragmatic TV.

http://bit.ly/Pk3z3

Jim Weirich expains how git (the version control tool) works from the ground up, and in doing so, builds a hypothetical system that sounds like what you are trying to do.

Interesting problem. Several things come to mind:

1) The Pragmatic tip "Keep knowledge in Plain Text" (fro the Pragmatic Programmer book, that also brought us DRY). You can argue whether XML, JSON, etc are considered 'plain text', but the spirit is simple - data is open when it is usable.

2) tools like diff and patch. If you make a change, you need to be able to extract that change from the whole and give it to other people.

3) Version control tools to manage the complexity of forking, branching, merging, and otherwise dealing with all the many little 'diffs' people will create. Git is an awesoe decentralized tool for this.

4) Open databases. Not just SQL databases like Postgres and MySQL, but other database types for other data structures like CouchDB, Mulgara, etc.

All of these things come with the poer to help address this problem, but come with a barrier to entry in that their use requires skill not just in the tool, but in the problem space of 'data management'.

The problem of data management, as well as the job to point to one set as 'canonical' should be in the hands of someone capable of doing the work. PErhaps there is a skillset worth defining here - some offshoot of library sciences?

I hink someone forgot what the 'U' in USB is supposed to stand for...

Yes, PHP is a heck of a lot slower on proccessor-bound tasks than C++. In a pure benchmarking contest, no doubt C++ will win.

But what about when both languages have to query a database (be it mysql/postgress/oracle, etc)? In this case, both are blocked on the speed of the database. a 15 ms query takes 15 ms no matter what language is asking. Facebook is not calculating pi to 10 gazillion digits, and it is not checking factors for the Great Internet Mersenne Prime Search. It is serving up pages containing tons of customized data. This is not proessor-bound... it is I/O bound both on the ins and outs of the database and the ins and outs of the http request. It is also processor bound on the page render, but the goal of this many machines is to cache to the point where page renders are eliminated.

Once a page is rendered, it can be cached until the data inside of it changes. For something like facebook, I bet a page is rendered once for every ~10 times it is viewed by someone. Caching is done in ram, and large ram caches take a lot of machines.

So lets look at those 30,000 machines not by their language, but by their role. We can argue the percentages to death, but lets assume 1/3rd are database, 1/3rd are cache, and 1/3rd are actually running a web server, assembling pages, or otherwise dealing with the end users directly (BTW, I think 1/3rd is way high for that.)

So 1/3rd of the machines are dealing with page composition and serving pages. If they serve a page ~10 times for every render request, then abtou 1/10th of the page requests actually cause a render... the rest are being served from cache. Those page renders are I/O bound, as in the example above - waiting on the database (and other caches, like memcached), so even if they are taking a lot of wait cycles, they are not using processor power on the box. The actual page composition (which might be 20% of the processing that box is doing), would be a lot faster in C++... So 10,000 servers, the virtual equivalent of 2000 are generating pages using php, and could be replaced by 200 boxes using stuff generated in C++.

So the choice of using php is adding ~1800 machines to the architecture. or ~6% of the total 30,000. Given that a php developer is probably 10x more productive than a developer in C++, is the time to market with new features worth that to them? I bet it is.

Had it really happened by then? If it hadn't been observed, wasn't the lander really a 'Schodinger's cat', although in a really really big box?

And does the lawyer offering this advice accept a retainer fee from his clients so that he can be on call for them?

24x7 support is costly in any business. The firefighter is not an apt analogy... Is he expected to work an 8 hour day and THEN be on call for fires?

And is he serious when he thinks a firefighter is paid for only the small amount of time he is out firefighting? If that were the case, I expect we would see a lot of financially insolvent firefighters-turned-pyromaniac in order to put their kids through college.

How do you *test* something like this? I mean, you build all this hardware write a bunch of code, and I guess you can see activity that might look like an EEG, but how do you know you haven't just created some elaborate noisy feedback system? How do you know you haven't created an autistic mouse? Short of giving it a simulated DOOM environment to run around in and chase laser pointers, what actually is it *doing* anyway?

I never thought unit testing would verge on philosophical questions.

Capistrano started life as a deployment tool for Ruby on Rails, but has grown into a useful general-purpose tool for managing multiple machines with multiple roles in multiple environments. It is absolutely the tool you will want to use for deploying a complex set of changes across one-to-several machines. You will want to keep code changes and database schema mods in sync, and this can help.

Ruby on Rails has the concepts of development, test, and production baked into the default app framework, and people generally add a 'staging' environment to it as well. I'm sure the mention of any particular technology on slashdot will serve as flamebait - but putting that aside, look at the ideas here and steal them liberally.

You can be uber cool and do it on the super-cheap if you use Amazon EC2 to build a clone of your server environment, deploy to it for staging/acceptance texting/etc, and then deploy into production. A few hours of a test environment that mimicks your production environment will cost you less than a cup of coffee.

I have tried to set up staging environments on the same production hardware using apache's virtual hosts... and while this works really well for some things, other things (like an apache or apache module, or third party software upgrade) are impossible to test when staging and production are on the same box.

So, the Universe abhors the creation of the Higg's boson so much that if we ever create it, it will time travel back to destory the machine that created it. Either that, or someone screwed up while assembling what might possibly be the most complicated machine ever built.

Whats that thing about simpler solutions?

Like an ID for a database record, the name should be unique, mean nothing out of context, and used only to look up a description of all the information you are trying to encode in it. What happens if the warranty info changes? What happens if you assign the wrong machine, move where it is located, or change some other fungible property (either through upgrades, or simply because you encoded the wrong info?). You don't want to have to go through machine renaming exercises, updating dns entries, etc. or have to live with the degredation of your naming convention.