On World of Warcraft's Network Issues 407
alphaneutrino writes to mention a C|Net article discussing some of the recent problems the World of Warcraft playerbase has experienced. From the article: "'Being a system administrator myself, I have some understanding of what goes on in a corporate data center,' said Evgeny Krevets, a sometimes-frustrated WoW player. 'I don't know Blizzard's system setup. What I do know is that if I kept performing 'urgent maintenance' and taking the service down without warning for eight-hour periods, I would be out of a job.' Blizzard blames some of the problems--such as the disconnection, for several hours on Friday, of players linked to several servers--on AT&T, its network provider. (AT&T did not respond to a request for comment.) "
$15/mo times six million users.... (Score:2, Insightful)
And _why_ are there any problems whatsoever?
Blizzard, I can guarantee this: if you spend $35 million per month on refactoring, hardware and bandwidth, all your problems go away. Guaranteed. I promise.
Re:wow (Score:5, Insightful)
Server preformance varies from realm to realm. I hadn't really had any issues until the last week or two when my server decided to drop 40 minutes into our 45 minute baron run, and then again in the BG's later on.
As someone else mentioned, I think they are still a victim of their own success. Sure it's been over a year since launch, but they were expecting 250,000 subscribers and got 6,000,000.
Re:$15/mo times six million users.... (Score:2, Insightful)
O.o (Score:2, Insightful)
The reason they're having so much trouble is because the integration with the AT&T to government monitoring station upgrades are taking too long.
AT&T: Keeping terrorists off WoW!
Re:$15/mo times six million users.... (Score:4, Insightful)
But, then again, I may also be an idiot... who knows?
Re:wow (Score:2, Insightful)
It's funny what people get used to. In the original EQ, patches were just a few hours in the morning, one day a week, unless something went wrong (which generally didn't happen, despite what the boards say).
In Horizons, another MMORPG, database lag was so bad that you could pick up an item and not see it in inventory for 10 minutes. You could run through an area full of monsters and not see one by the time you were through, because the client couldn't build them fast enough.
If you're shovelling out $15/month for a service, about what you pay for cheap telephone service, you'd expect it to work when you wanted it to work. Like I said, it's funny what people lower their expectations to.
No wonder Bush is still president (snap!)
More Crafty (Score:5, Insightful)
And that AT&T is exploiting them, marketing a new "premium service/support" contract by letting them go down.
I can't wait until WoW has to pay AT&T (and its handful of competitors, if they get rid of the SPF) the extra "premium tier" routing fees, once the telcos market their "nonneutral" Internet. Because a world of angry Warcraft players jonesing for their fix will be a nice gift for telco suits just trying to make it home from work.
System Administrative Failure (Score:2, Insightful)
The common system that is administered in a corporate environment does not have thousands of users connected at once all requiring huge amounts of bandwidth and processing time 24/7. That is not to say that systems with large wear and tear don't exist, or that systems with such huge numbers of users don't exist, or even both. What I am saying is that Blizzard has to administer two or more dozen server clusters being continually accessed 24/7 by resource intensive users (save for a usually brief repose on Tuesdays). Unless you work for Google (and even then) there's no comparison.
This isn't to say that we shouldn't raise our eyebrows at the pervasiveness of the problems WoW has. However, I keep seeing the same arguments thrown around about how Blizzard gets $15 x 6million every month (not entirely true because A) there are less pricey payment options and B) something around 1/3 of those players play in China on Asian servers whos subscription plans would hardly be purchasable were they $15 a month), how at 'my company' things work this way, etc. etc.
Blizzard should be called on to answer for why their servers haven't been made right as rain after a year or more after release, but it should be in the context of legitimate complaint and not any of this throwing around of overused and hardly consequential arguments.
My account expired yesterday... (Score:3, Insightful)
I know it is tough for Blizzard, but as a customer I have been the one paying the price for that so far, from now on that cost is Blizzards again. At least for the time being.
Ill communication (Score:5, Insightful)
Seriously, a little timely information goes a long way. Yes, I agree that the downtime they have is absurd; consider that *every Tuesday* the game goes offline for *six hours* of maintenance. That's *planned, scheduled* downtime, folks, so that *alone* means they aren't even attempting to have greater than 96.4% uptime, and I can't think of another commercial service for which you pay a monthly fee where that would be even remotely acceptable; if your cable or your phone just plain didn't work for 6 hours every Tuesday, heads would roll. Then things just get asinine when you factor in all the spontaneous, freewheeling, unplanned downtime as well.
But know what? I'd feel a lot better about it if, when something shits the bed, or goes tits-up, or whatever colorful metaphor you'd use to describe a server-killing technical problem, Blizzard would tell us, promptly, as they receive the information themselves:
1. We know there's a problem.
2. We know what the proglem is.
3. Here's what we're doing to fix it.
4. Here's when we expect it to be fixed.
5. Update as old information is obsolete.
They don't do this. A few hours after something happens, you might get some of the above information. Or you might not. Usually, it's the latter.
Comment removed (Score:4, Insightful)
Re:wow (Score:2, Insightful)
Its all a matter of money (Score:2, Insightful)
Yeah, but the difference with WoW is the money. When eBay, Amazon.com, and E*Trade have outages they are losing money. When WoW has an outage they don't lose a dime. Only thing they lose is the 1 or 2 players who get frustrated and abandon all the 'work' they put into their characters and cancel their accounts. Blizzard still collects your $15 every month, outage or not. No $$$ incentive for them to provide good service. I thought they were a good company, but my opinion of them is changing drastically. Diablo 2 and Warcraft 3 are both Blizzard games you can play online for free, they are more stable then WoW. Something is wrong here!
Re:A typical week on Mal'Ganis (Score:1, Insightful)
I create a new character on a 'new/low' population server (Malygos). My ping is around 20-30ms. Everything is silky smooth.
After two months, my ping is 100-120. I start to see 'stuck crouching' bugs, instant cast spells are now 2-3 seconds, and selling things to vendors hangs for 5-10 seconds.
After four months, my ping is 200-220 on average. PVP is now spiky and aggrivating as people warp around the screen on occasion. We're now a HIGH/FULL rated server.
In April, Blizzard raises the 'acceptable' ping range (green) from 0-100 to 0-200, and the light green range to 300. They also make the ping meter average out over 3-4 minutes, completely smoothing over all but the worst spikes.
Nice fixes there.
In August, the ping meter is no longer reliable - setting up a network monitoring program shows huge spikes of 1000+ and packet loss of 2-3%, yet the ping meter just hums along at 200. Gameplay continues to decline and you get occasionally booted unexpectedly from the game. Queue times start up on the server, so now getting booted means a 5-10 minute wait. Tuesdays (patch days) are now more or less unplayable. Sundays and Mondays before maintenence are getting close to unplayable.
This cycle continues until your character you've played for half a year is worthless due to lag. You can reroll on another new server to watch the process start all over again... I've done this four times already (Eredar, Ursin, Gurubashi)... which is what people tend to do because they're addicted.
Re:$15/mo times six million users.... (Score:3, Insightful)
Do you really think that they are currently depending on a bunch of hacks that are coding in Visual Basic and their entire network knowledge consist on knowing the correct sequence to crimp in a RJ-45 ethernet connector?
Are you proposing a full rewrite? Are you serious! Including testing, how long would that take? What you are proposing is pretty much that they scrap everything and start from scratch.
Same problem in Germany (Score:4, Insightful)
Re:Code patches? (Score:2, Insightful)
Re:What I love about patches and hotfixes... (Score:4, Insightful)
Real code is complex, and generally written as a massive matrix of inter-related side-effects causing things to happen*. When it gets written, the entire matrix is designed, intended, documented, and understood. Two years later the guys working on the code have no clue about the matrix of side-effect driven code, no clue about the complex set of business factors driving the technical aspects of the code (and by business factors, in a MMORPG I mean things like class X has bad faction with everybody making it more difficult for him to start out, but in return for overcoming that challenge has more powerful magic later in life - stuff like that) and when they are making a change they go in, find the one line of code that looks like what needs to be fixed and just change it without knowing all the places that change will ripple back to, invisibly, via the side-effect matrix.
A technical phrase to understand here is 'globally scoped variables' - and another one is 'design intent' - and as the current set of hacks don't understand the ramifications or scope of either, this is what happens.
Footnotes
* I didn't say it was a good idea. I just said it happens.
Re:wow (Score:5, Insightful)
The controlling factor for their server performance should not be the total number of subscribers, but the number of subscribers per realm, and Blizzard has complete control over that number, because they can mark a realm as "full" and disallow logins/signups. IOW, as you know, those 6,000,000 people are not all playing in the same game at the same time.
It should be possible to make the realms completely independent, so that this just becomes a matter of horizontal scaling, and having hardware/systems monkeys roll out new realms via some standard operating procedure.
Unfortunately, based on the rumors I have heard, Blizzard has chosen to tie a bunch of stuff together. For instance, the common web forums use the characters from all the realms (the web forums know about your level 23 mage), they have a single set of auth servers, it's not clear that the item databases are not shared between realms, and so on. This is sort of sad, because it's not like Blizzard are the first people to roll out an MMORPG.
Now, some might argue that tying some of this stuff together makes for a better user experience. However, when this entanglement leads to downtimes, one could make the argument that it's not worth it.
Anyway, my point is not to bash on Blizzard; I'm sure they've made some difficult design decisions correctly, and some difficult ones incorrectly. My point is that "we have lots of users" is not a good excuse when you have a service that lets you divide those users into sub-populations, and that there are probably architectural improvements they could make to improve their scalability. The real question is whether they have competent and experienced systems engineers to help them make those improvements, and whether management is committed to supporting them.
Anyway, so much for pre-coffee ramblings....
Re:$15/mo times six million users.... (Score:3, Insightful)
ANY company that makes a significant portion of its money from internet sales should have multiple providers for EVERY public-facing server. I'm not saying that every machine should have multiple NICs, but they should have their servers connected to beefy network equipment that can switch all traffic to working providers the moment one provider has problems.
In this case, they can even control the protocol and contents of the packets, so they can't even blame protocol limits for the problem.
Re:wow (Score:2, Insightful)
Then Blizzard should not have distributed 6 million copies of the game. They've brought this upon themselves. Open beta consisted of a very small user base (relative to what it is now). So the kind of resource pressures they face now were never realistically tested pre-release. So really, the first (and it seems, current) adopters of the game became part of a second beta test of sorts. And I'll grant Blizzard that they were caught off guard but they've had almost 2 years to get things right and they still haven't managed it. And regardless of their known problems they continued to distribute more and more copies of the game. They should have shown a bit more restraint after initial release but they went all-in and now we're stuck with what we have.
The big problem is that customers, especially those who've been around since release, will be reluctant to leave even if the service drops further. Reasons being addiction and the loss of investment on the time spent playing the game.
I think WoW could go completely offline for a week and come back with a very minimal loss of its user-base. They don't have the motivation to do better.
Comment removed (Score:3, Insightful)
Re:A typical week on Mal'Ganis (Score:3, Insightful)
Re:$15/mo times six million users.... (Score:2, Insightful)
I was one of the "never going back to any SOE product" EQ burnouts, but in both EQ2 and SWG the corporate overlords have come down on the "screw the players, we hate them" mentality with a vengance. Major staff changes happened, major game changes are underway (SWG is almost a completely new game), and one gets the sense that, finally, they *get* it.
Here's hoping, anyway.
Re:Ill communication (Score:2, Insightful)
Many of these are much easier said than done.
1) Identifying the problem is the first step. Determining that a slow login is on the customer's end or their end is a task in and of itself.
2) Again, this takes time and is closely associated with #1 above. knowing that you have a problem and pinpoint the cause are 2 different things.
3) Not really the customer's business and very closely tied to #2. A lot of the time, determining the problem on a network gets you 90% of the way to correcting it.
4) Same as #3. Determining the problem is the most time consuming part much of the time. If you don't know what the issue is, you don't know how to fix it yet. If you don't know what it's going to take to fix it, logically it's not possible to give an ETA that's worth anything. It's also not really in their interest to say "We'll have the login problem fixed by 8PM" because that just encourages people to pound the servers at 8PM creating a whole new set of issues. Having the infrastructure to support 6M customers is different than having the infra structure to support ^m customers that all want to login in at the same time
5) Yeah, this would be helpful provided that you're not just pushing back ETAs replacing the last inaccurate one with a new inaccurate one. This just gets people frustrated.
Re:$15/mo times six million users.... (Score:3, Insightful)
They've had nine months. They should be having a baby every month now.
Downtime is either being caused by poor design, hardware/software limitations, or bandwidth limitations.
These are not things that are unknown, or uncontrollable.
If this project is too complex to get a firm handle on the problems, and the work force can't be scaled to meet the demands, then your only avenue for relief is to scale back the complexity.
Otherwise you're admitting that you are managing the project poorly.
For those that don't play, keep in mind that this is an ongoing, crippling (ie, showstopper) situation. This isn't occasional short downtime.
It's as if every time there's a crash they have to run a WoW style FSCK on the world image before they can bring it back up.
I know I'd be much happier if they at minimum
1) Acknowledged any downtime immediately (ie, if I can't get on then there should be a status message about it - not four hours later)
2) Described, briefly, what stage they are at in troublshooting (ie, is the problem understood? Is the fix being implemented? Has anyone even been notified?)
3) Gave a time frame. (Will be accepting logins within the hour, within 5 hours, within 24 hours.)
I don't know, maybe everyone else has copious amounts of free time, and the ability to play whenever they want. I have to schedule time for myself to play WoW. When I can't play I essentially lose that opportunity - I can't just swap things around.
At least throw us a bone and give us the information we're all begging for, since you certainly aren't going to give us back our time or money.
-Adam
Re:Nothing new for MMORPGs (Score:2, Insightful)
I've seen this argument before, and I'm completely baffled by it. If Blizzard ordered to have 1,000,000 copies of the game printed and distributed to retail locations, how can they be surprised when 1,000,000 people try to sign up?
Comment removed (Score:3, Insightful)
Re:Monthly fee (Score:3, Insightful)
The interesting thing is that with an MMO like WoW (not just WoW, there are others similar) you actually get quite a bit of content in the lower levels. You then run into this brick wall. It really starts with the tier 0 sets in the level 55-60 dungeons. You get to spend a couple hours for an 8% chance for an item that you might have to roll against another player to get. The first couple of runs through an instance can be good fun. The 20th run usually is not. And that is just a prelude of the still to come ZG, MC, BWL, AQ experiences of the end game.
I had 6 out of 8 pieces of tier 0 gear and had done some ZG runs when I decided to quit. The game that to me was great from lvl 1-60 (ok well maybe 1-55) offered nothing but huge, boring and repetitive time sinks.
I really enjoyed exploring WoW and think it's a great game - but I'd rather be exploring new content in some other game than going for my 30th run in Undead Strath hoping that the Baron will be wearing my pants tonight.