Looking Inside the Second Life Data Centers

An anonymous reader writes "InformationWeek looks inside the data centers that power the game Second Life. Tidbits from the article: The software architecture is an extension of the virtual world metaphor of Second Life. At any time, it's possible to walk into one of Second Life's two data centers, pat one of the rack-mounted servers, and say that particular server is running virtual New York, or San Francisco, or ancient Rome, and imagine itty-bitty people and buildings inside the 1U rack-mounted servers. Linden Lab, which develops and maintains Second Life, runs 2,000 Intel- and AMD-based servers in two co-location facilities in San Francisco and Dallas. And, contrary to widespread belief among Second Life users, Linden Lab has not decided whether to open-source the Second Life server software."

OpenSim?

"Linden Lab has not decided whether to open-source the Second Life server software."

I dont think it matters too much, the opensim project has been making amazing strides using the BSD licensed libsecondlife code as a base. http://openmetaverse.org/wiki/OpenSim [openmetaverse.org]

What exactly does this mean?

I'm not too into server architecture, but it sounds like it's a good system to me. Could someone explain why this is so "new" or different?

Well, it isn't a "good" system at all

It works, up to a point but it is extremely limited.

If you read the article you will have noted that an area in the game is run on a single processor. That makes it fairly simple to grow, more areas == more servers.

But it is a bit like handling multi-tasking on your PC by adding more cores for every task. Run your OS, 1 cure, run a music player another core, run a game, another core, run a virus scanner, another core.

This is NOT the way things are done and for three reasons.

First it is wastefull, an empty area (no players) would still be using a full processor, granted probably a light one but it would be like having one Pentium4 cpu dedicated to running your mp3 player, even the cheapest available is going to be overkill.

Second it is limited, you can only use 1 cpu and they are still limited in how fast you can go, worse each speed increase is going to cost you more and more. So an area with lots of visitors will be unable to scale.

Last is that areas are seperated, you have to move from cpu to cpu as you move areas, this means transferring a lot of data even if you go from one desolate area to another.

Imagine if an ISP had every website on a single CPU box and that is the only option. Wastefull for small sites, not powerfull enough for large sites and a nightmare to administrate.

So why did they do it?

Well, it is relativly simple to setup. You don't need loadbalancing for instance or dynamic scaling. Customer simply buys a server space from you and that is the their server. It should in theory also be fairly robust, one cpu/server crashing won't really affect all the others. In a cluster setup one bit going down CAN (doesn't have to but it seems like it in MMORPG terms) take everything with it.

it is also cheap, they can use stock hardware buyable from any cheap box maker. Blizzard and Sony had to cough up some serious cash long before they could even open their game to get their servers running.

It is the reason why today the majority of hosting providers still work with crappy intel/amd boxes and not virtual servers on proper sun/ibm or some such hardware. It is cheap and you can get started with just one desktop PC (I seen server farms that had racks specially designed to house desktops, not racks).

More traditional setups for MMO's are to have clusters, each cluster is made up of a combination of hardware setup to serve a particular area. The advantage here is that you can more easily upgrade a cluster to handle a bigger load from an area. There are limits but more or less you can simply plug in more hardware to handle a high load. Offcourse such hardware is going to cost you.

The software for it is more complex to build and in all it is just more costly BUT in the long run more flexible.

Linden Labs had (still doesn't) have anywhere near the resources of a SOE or Blizzard. Their system worked for them but by now they are feeling the pinch as some areas just can't handle the load.

Their advantage is that customers themselves pay for the servers directly, so anyone with an underused area is wasting their own money, not Linden Labs. Same as when you buy a dedicated super server to serve you knitting club photo's. Your money your waste.

By the way, the above is based on an extremely old in depth article, it could well be that nowadays a sim (area) can use more then 1 cpu, but back in the day it couldn't

Game?

Second Life isn't a game, it's an advertising medium. Nothing more.

Re:Game?

You can create an account and run around the secondlife universe all day for free. You only have to pay if you want to own virtual land within SL.

Re:Game?

Yeah, it's amazing.... I can't think of any other advertisement medium where viewers pay to see advertising. I bet I could make a killing with such a service - I'm gonna start my own business. I think I'll call it Able Mellowvision.

OSL?

I think, instead of using Second Life as a base, they should have started from scratch and fixed some of the 'issues' with Second Life.

You can't use anything but primitives. Making a non-simple object often requires more polys and ingenuity than it should. A cowboy hat, for example.

Proprietary scripting language. Going with Lua (more popular) or Ruby (my choice) would not only be easier to use, but would also let budding geeks learn a good language. SL is implementing .NET, if I remember correctly, though. Not bad as a third choice.

Texture maps, shaders, etc, etc. SL supports no advanced graphical features.

I'm sure someone will say 'get off yer lazy butt and do it yourself', but it's obviously not that easy. I don't have the time, money, or skill to create an entire virtual 3D world that is user-scriptable. And gathering a team of those who DO have those things is tough on a from-scratch project.

Residual self image

At any time, it's possible to walk into one of Second Life's two data centers, pat one of the rack-mounted servers, and say that particular server is running virtual New York, or San Francisco, or ancient Rome, and imagine itty-bitty people and buildings inside the 1U rack-mounted servers.

"No...I don't believe it...let me out! I want out!"

OMG

There's that thunderous "pat-pat-pat" from the heavens again...time to sacrifice more AOL CDs...

Pictures

Why do all of these "inside the data center of ..." never have pictures? Everyone knows geeks needs something to drool over and fantasize with.

Re:Pictures

Here's a couple recent linsk for data center photo tour geeks: Info Week just had a slideshow of photos from inside the Sun Blackbox [informationweek.com] portable data center, while C/Net offered a look inside LucasFilms' Death Star data center [com.com] in San Francisco.

FOSS details from TFA

Second Life servers run Debian and use MySQL. They are transitioning to use Mono as a scripting language (from their own scripting language, which apparently isn't working out so well).

Which is nice. However, not open-sourcing their server code is somewhat disappointing. Oh well, at least the client is open, someone else can create a FOSS server if the interest ever arises.

Re:FOSS details from TFA

http://www.libsecondlife.org/wiki/Main_Page [libsecondlife.org]

"The libsecondlife and libsecondlife-java projects both provide the additional flexibility of having a BSD license,"

Your definition of 'not FOSS' is radically different from mine, then. BSD License is just about as open as you get.

Wheres the server pr0n?

Am I the only one that reads these titles that say "A peak inside X's datacenter" and go "OMG Datacenter pics!!!!!!"? Only to be disappointed by a text article with no pics? I DEMAND MY SERVER PR0N!

Land prices are a scam, artificial land shortage

I did try Second Life, I wanted to try if I could have something similar to a web page into the virtual world.

I discovered that land price is artificially high, what happens is that "landlords" buy "islands" and then resell small plots at high margin.

Let me do a few calculations (correct me if any of the assumptions are wrong)

• An island is driven by a server and an island maps 65535sqm square meters
• A one year resident pays about 70 US dollars and once upon a time you could have a 512sqm plot with it (it means that they had the First Land policy)
• If we divide an island by 512sqm we have about 127 "plots", i.e you can put 127 residents, each paying 70 US dollars.
• It follows that each "server" can host customers paying about 9000 us dollars EVERY year

Current land costs ten time as much as that, at least, and you find weird plots or space for rent ant HIGH prices (compared with the cost of servers). This seems to me a scam, I have even blogged about it, hoping that someone notices. [engidea.com]

I did write to Linden about it, but I have yet to receive a reply [rt.lindenlab.com #631998], what irritates me is that current policy just makes Landlords rich doing basically nothing, it does not seems fair at all to me.

Regarding my original plan, I have realized that it is not enough to have a "shop" there, you also have to staff it. So, the cost of having a shop in second life is way more that a website !

They might want to upgrade...

Over the last week I've taken my fifth visit to SL since it began. Although the interface has changed a smidgen, and there are now partially fleshed out tutorials nothing has really changed.

Avatars still look like very old Poser characters. Animations for things as simple as walking are poorly done. People look like skittering wind up toys.

Performance is very, very poor, as always. Moving even a couple of virtual feet triggers up to several minutes of lag while thousands of objects load, usually in teeny bite size primitives.

And as usual, within a couple of minutes leaving the tutorial area I was hopelessly lost and unable to find anyone to talk to. :)

Second Life has always been nothing more than an experiment in marketing. I don't understand how anyone could possibly spend more than a short while in there before becoming completely frustrated with the hideous performance.

Security!

"At any time, it's possible to walk into one of Second Life's two data centers, pat one of the rack-mounted servers..."

Who says they'll stop at patting? What if they cripple the servers?! It sounds like Linden need to hire some security.

Second Life in the Toronto Star today

(Virtual) reality bites [thestar.com]

"With the encroachment of big corporations, residents of Second Life say the online world is becoming second-rate, writes Murray Whyte"

(Not sure if that's the same title used on the dead tree version I read this morning.)

1 server per "place"?

is wyoming running on a 386 in a closet somewhere then?

What about storage?

All they mention is that they have 34TB of storage... Is this locally attached / internal disk? NFS? SAN? FibreChannel? iSCSI?

Storage... it aint sexy but you know you want it...

Yup

What a phenomenal waste of electricity. How many kilotons of coal does it take to power this pig? I just hope we can all move into 2nd life when our own planet starts writhing around.

location location location

Tell me their data center is on the 13th Floor. =)

Inter-server communication

FTA: Avatars near the edge of one sim need to be able to look over and see activity in the next sim, just like people in the real world can see activity going on in the next lot of land.

What would happen if everyone moves their avatar to the edge of an area and look towards another area? Would this bring down the communication between the two servers?