Dow Jones Plunge Fueled by Overwhelmed Computers 215
cloudscout writes "The Dow Jones Industrial Average dropped over 400 points today. While there were various valid financial reasons for such a decline, some of the blame is being placed on computer systems that couldn't keep up with the abnormally high volume at the New York Stock Exchange and the resulting tremor as they switched over to a backup system."
Blaming? (Score:4, Insightful)
Humans do, at least in designing, manufacturing and sizing computer systems.
This one seems to me like blaming at a knife once you cut your fingers.
Re:Blaming? (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2, Informative)
Re: (Score:3, Informative)
Re: (Score:3, Informative)
Re: (Score:2)
Decline, Before 1:00 P.M., Before 2:00 P.M., 2:00 - 2:29 P.M., 2:30 p.m. or Later
1250-point Decline in the Dow, 1 Hour Halt, 1 Hour Halt, 1/2 Hour Halt, No Halt
2500-point Decline in the Dow, 2 Hour Halt, 1 Hour Halt, Close For Day, Close For Day
3700-point Decline in the Dow, Close For Day, Close For Day, Close For Day, Close For Day
Oh yeah
Trading-collars, which restrict
Which computer and OS are at fault? (Score:2, Funny)
I hope that the answer is not "Linux on top of HP hardware."
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Funny)
Re: (Score:2)
You'd have to have an independently-evolved AI robot bombard the computer with alpha particles.
Tell that ... (Score:2)
Re: (Score:3, Insightful)
Sure they do. Bad ram, overheated components. If computers didnt make errors, you wouldn't need CRC checking - to name but one common error identifier.
Lets try and think more, go for frosty pist less. Ok?
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
NYSE: Microsoft (R) powered (R) (Score:5, Funny)
Re:NYSE: Microsoft (R) powered (R) (Score:5, Interesting)
all of the machines you see in pictures are dumb terminals connected to a number of master unix systems. they are configured for that special keyboard set used for deals and also touchscreen inputs. however, some (not all) can be booted into windows for personal use or to use firm-specific software.
a side note -- the monitors used by NYSE (you can see them in the pictures on CNN) are ridiculous. huge size, almost perfect 180 degree visibility from side to side and top to bottom, touchscreen, and dust/fingerprint repellent. pretty nasty stuff.
as far as the backends go, the standard system and the backup system are kept running in two locations -- on site, and in a backup location outside of manhattan (not to hard to figure out where, if you try). one can function without the other, on both counts. The QA done to ensure validity is INSANE. Nothing can even be brought out on to the floor without making sure it won't corrupt any of the systems (including the bluetooth system that has become the lifeline of the trading floor -- largest (in size and traffic) private bluetooth network in the world, when I was working the floor)
Re: (Score:2, Informative)
The machines that calculate the Dow Jones Industrial Average are IBM hardware running Unix.
SQL Server running NASDAQ: http://www.windowsfs.com/eNews/tabid/112/articleTy pe/ArticleView/articleId/933/Securities-NASDAQ-Mig rates-to-SQL-Server-2005.aspx [windowsfs.com]
IBM Unix Machines running NYSE and calculating the Dow Jones, as pointed out by ano
I have to wonder if this is spam related? (Score:3, Insightful)
Re:I have to wonder if this is spam related? (Score:5, Informative)
Apparently the system that computes and displays the current Dow couldn't keep up with the systems that process the transactions when the number of transactions became very large. The display system caught up a bit later making it *appear* that the market at suddenly dropped something like 250 points in a few seconds.
In reality, the decline was fast but steady. It was just the exchange's version of "Damn lag!"
Re:I have to wonder if this is spam related? (Score:5, Interesting)
Re: (Score:2)
Re:I have to wonder if this is spam related? (Score:5, Interesting)
if you actually bother to read TFA - you'll find they don't blame the computers for the drop. They blame the computers for creating the perception of a cliff rather than a steep slope.
This is actually something fairly important to consider as more and more of our life interfaces with displays mediated by computer. (Even down to the mundane. I've discovered the digital controls in my oven seem to use a time weighted average - which works fine in convection mode, but it enlarges the deadband in normal mode.)
Re: (Score:2, Informative)
Re:I have to wonder if this is spam related? (Score:5, Informative)
The computer problems experienced were really just a lag between the DJIA being calculated and the massive volume of trades being made. Individual stock prices were being reported correctly but the index wasn't keeping up. When the computers caught up they did it over a single minute dropping about 300 points while in reality by the time the index caught up the market had started to rebound a bit. All of the value stock buyers saw the deals becoming available when the landslide hit and started buying a bit. Kind of like today, the market is rebounding because many are looking at stocks that were overpriced yesterday and thinking they are cheap. It's not really as big of deal as the press makes it out to seem. It's not like the '87 crash where 500 points was like 20% of the market. 500 points off th dow is under 4%.
Another trigger for the sudden decline could have been the headline on The Drudge Report (linking to the New York Times article by the same headline) stated that Greespan predicted an imminent recession when his words were as they always have been and that people should be carefull because the economy has been growing for longer than the average growth cycle by about 12 months. Greenspan didn't say anything about a recession being imminent.
This is why... (Score:4, Funny)
The Dow may go down, but a pizza is still as tasty hehehehe.
I wonder how much of this load is due to low volume day trader movement?
Tom
Don't see how they could. (Score:3, Insightful)
I would guess, virtually none, since they're by definition low-volume?
This blaming it on computers seems mostly a red herring. The markets in Asia (particularly Shanghai) tanked, and as a result, the markets in the US tanked, because companies in the US are heavily invested in China.
I think the only lesson here, in case there was anyone left who didn't get it, is that we all float or sink together. For better or worse, the US has tied i
Re: (Score:2)
Do you really think the traffic is higher if you move more units of stock? What I meant is there are probably millions upon millions of people moving $200 worth of stock here and there. Versus the tens of thousands moving significant quantities.
Tom
Re:Don't see how they could. (Score:4, Interesting)
I think that it's more of a symptom and less of a cause. The cause of the market movement was in Asia; that made people sell, people selling caused the DJIs computers to suck. Now, perhaps the DJIs computer slowdown, and consequent large jump when they fixed the problem and got the backup running, caused more people to sell, but this seems specious. The slump was already in progress by the time that the computer slowdown occurred, because the slowdown was driven by high trade volume.
So my point is mostly that I don't see how it matters, really. People are looking towards computer glitches as the cause for the 3% drop in the market (or whatever it was), and that's just not true. The computer glitch might have made the drop look worse, or more precipitous, than it actually was at one point during the day, but it didn't cause or really drive it in any significant way. Even if the DJI folks' computers had worked perfectly, the market would still be sucking. In fact, computerization and the consequent flow of information is what links markets; it's only in the last few decades that the Asian and US markets have felt each other's pain so closely, so in a way, you can blame the computers for working too well in general, when you get these domino-effect deflations.
Re: (Score:2)
No, what really happens is that a lot of the big money has automated trading systems. These automated trading systems look for key criteria like quick movements in stock and buy or sell to make money (or avoid losses) on the trend. When a big movement happens in a short period of time, it creates a feedback loop where big movement ca
Re: (Score:3, Insightful)
"Load"? That's exactly what I was thinking (Score:4, Insightful)
Re: (Score:2)
You know this, I know this, some others may know this, but the problem is people let themselves get caught up in it. Very few people really give a damn about gay rights [for example, one way or another] but they'll sure as damned have an opinion about it [usually supplied by the media or some ass on the medium],
People know the news is fake, at some level, they just *need* to believe it. So long as the bad guy is someone I don't understand, and all those
Re: (Score:2)
I'd also point out that a one-day blip, even a big one, doesn't really mean very much. "Savers" win and lose in the stock market by long-term trends, not the short-term.
Re: (Score:2)
See... (Score:5, Funny)
Chaos theory, anyone? (Score:5, Insightful)
2. Market starts to waiver
3. Other parts of the market see this tremor so market waivers a lot
4. Panic ensues
5. Indices drop 10%
6. a pension company goes bust
7. my grandpa doesn't get to eat.
The last few steps are somewhat hypothetical, but still. The stock market must be one of the most immediately visible examples of chaos theory kicking humans in the nuts.
Peter
Re: (Score:2, Funny)
Your Nobel Prize is in the mail. Don't call us. We'll cal you.
Re: (Score:3, Funny)
Of course, the usual moron response to this is to say "why don't we just kill all the butterflies in Asia?"
Peter
Re: (Score:2)
"A butterfly flaps it's wings in Asia , and the dow drop 400 points in the US... brilliant."
More likely a disk drive started to fail, but was able to recover. They can do upward of several hundred retries over 5 to 10 seconds before they return a fatal error.
If the disk drive completes the operation after just a couple of dozen retries, they'll keep on going without reporting an error. If no error is reported back to the controller the raid functionality usually won't kick in (depends on the
Re:Chaos theory, anyone? (Score:4, Insightful)
At least, that has been my observation. I can't WAIT for leveraged index ETFs... come on, ProFunds!!
Oh, and your Grandpa's pension would not go bankrupt over a panic event. That's absurd.
Of course, anyone who relies on pension companies for retirement has bigger problems...
Re: (Score:2)
I can't help but think the same thing was said during 1929 and then again in 1989.
Re: (Score:2)
I wasn't trading in 1989, but I don't remember stepping over starving retirees on my way to school. I think kids these days know that they should keep most of it in index funds when they are young, then move it to government bonds when they are reaching retirement. To rely on a single company that relies on the stock market
Re: (Score:2)
Saving has become very, very simple recently. There are now "target-date" or "life cycle" funds that divide your investmetns into asset classes appropriate for your intended retirement date. Simply put your savings in the date closest to when you plan to retire, and it automatically diversifies for you and keeps your appropriate asset mix given how much time you have. They're available in most 401(k) plans (though some, *cough cough* assume an older wo
Re: (Score:2)
8. . . .
9. Profit!
Re:Chaos theory, anyone? (Score:4, Insightful)
1. Computer switch-over is a bit slow
2. Market starts to waiver
3. Other parts of the market see this tremor so market waivers a lot
4. Panic ensues
5. Indices drop 10%
5a. If your investments are diversified, you will survive when the Indexes drop 10%. This is especially true for long-term investments.
6. Buy low.
7. Wait for a while.
8. Sell high.
9. Profit!
10. Enjoy your retirement.
I'm actually suprised (Score:5, Insightful)
I'd say that someone, likely the one in charge of the IT budget approval, keeps tight purse strings. Of course, he's not the one getting reamed, it's the CIO and his crew who are taking the blame even though they have repeatedly requested the funds to improve the system. Just speculation, but likely spot on.
Just another piece of ammo when I start a new job and demand a reasonable budget.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
The NYSE has historically been behind the curve as far as capacity to trade is concerned. I don't have links to back it up, but i recall a TV program that detailed how, before electronic trading was introduced, the exchange had to be closed during the week to allow transaction processing to catch up with the previous week's trading.
Re: (Score:2, Informative)
Just one stock for instance, APPL will have millions of transactions in a single days trading... Including just trades and quotes you can see close to 100Mb of activity for a single symbol in a day.
We are constantly trying to increase capacity, but we're near a point where the only meaningful upgrades for capacity plannin
Re: (Score:3, Insightful)
Put out an RFP for someone to design a cluster based system that will perform the same functions the system currently does, but using OTS hardware in a massive single or distributed cluster configuration. Perhaps they use virtualization and have one VM for every symbol... then purchase enough hardware that only the least active symbols share machines w
Re: (Score:2)
Generally it *is* ahead of things at peak capacity - but there was a massive spike yesterday that briefly exceeded peak capacity.
One of the problems Wall Street has been dealing with for decades is a tail chase with itself - every time they up their computer capacity, trading volume expands to consume the capacity. What was shortly bef
The Highly Reliable Times (Score:2)
Highly Reliable Indeed !!!!
Just like the computers in 1929! (Score:2)
wonder what the reactions would have been like if a "computer glitch" knocked the thing up 500 points instead of down.
Re:Just like the computers in 1929! (Score:5, Informative)
Um
You mean like what happened two weeks ago? [cnn.com]
Re: (Score:2)
Fueled nothing (Score:4, Informative)
Traders still bought and sold stocks at their real value in real time. The calculation of the sum of their activity was the only thing delayed.
Re: (Score:2)
They were selling based on the false impression that the market was not down enough yet.
Re: (Score:2)
Re: (Score:2)
"Some of the books froze up," he said, referring to the systems in which traders place their orders. "You couldn't really trade. You couldn't really make sales." He said orders appeared to become backed up. "Once they unfroze the Dow fell."
It sounds like at least some of the trade processing systems rely on the DJIA calculator in some way. Perhaps they query it to report the DJIA at the time of the transaction or some such. So trades were being processed but the feedback mechanism was fro
not quite (Score:5, Informative)
It's voodoo (Score:5, Insightful)
Imagine a series of database transactions, with each step getting queued up and waiting for the system to finish processing it. The actual DOW number reflects fully completed transactions, but not pending transactions that might impact the outcome. This is probably a good thing, as a transaction might end up being rejected, so you only want to show the outcome of completed transactions. Once the backup system came online, the transactions quickly finished being completed, resulting in the dramatic drop.
The amazing thing to me is that the system is robust enough that transactions can survive the loss of their main computer system and bringing up a secondary one. That's database, networking, and coding voodoo, all wrapped into something pretty awe-inspiring.
Re: (Score:2)
Pretty much the number one consideration when designing a technical architecture for a mission critical is to guarantee that no transaction is lost.
Such an architecture is not too hard to design.
The really hard part is to make one that can process millio
Re: (Score:2)
With todays databases like Oracle losing a translation just doesn't happen. (for the most part anyhow) What is cool is to have a fail-over that keeps in sync with the production DB, will fail-over, run the logs, and come online in mere minutes with a system this size and with this load. No matter what, it's impressive even if it's effect was visible to the use
Re: (Score:2, Funny)
Re: (Score:3, Funny)
It's an entirely different kind of computing.
Re: (Score:2)
How is this in any way awe-inspiring?
It was designed to do this. It was implemented. It worked.
You answered your own question!
Real Cause? (Score:3, Insightful)
Re: (Score:2)
It might seem somewhat simplistic, but since 'the market' didn't know it was in free fall, 'the market' couldn't correct.
The lack of information kept people from buying while stocks were steadily slumping, China or not.
Dropping prices versus dropping data (Score:5, Insightful)
The Dow did NOT drop 200 points in minutes, the data simply caught up with the drop that had already occurred.
Re:Dropping prices versus dropping data (Score:5, Interesting)
Re: (Score:2)
Re: (Score:2)
Because a lot of people don't know the difference.
Cause Was China (Score:2, Redundant)
Editors? We don't need no stinking editors. (Score:5, Informative)
Twilight Zone: Tom Clancy (Score:5, Interesting)
Re: (Score:2)
IBM backend w/ Linux Workstations used here? (Score:3, Interesting)
http://www.internetnews.com/bus-news/article.php/
Where by various financial reasons... (Score:2)
Other contributors were low durable goods orders and Greenspan's warning on monday of
Not just Dow Jones (Score:2)
Trying to access my account at Fidelity is also slow and with intermittent failures today and yesterday.
The "snowballing" is not in the prices moving too much up or down, but simply in the increased activity as a lot of people are trying to check their accounts and trade in and out of positions...
What system? (Score:2, Redundant)
Re: (Score:2)
how does this say anything about the systems at nyse? nyse was flooding dow jones with too many messages. nyse didn't have a problem - dow jones had the problem. it seems like a lot of folks posting to this story don't know the difference (no - i'm not new here).
i also don't see why t
Hal? HAL!!! (Score:4, Funny)
Damn it! (Score:2)
PGA
Oblig Futurama quote (Score:3, Funny)
The Dow's technical glitch, dissected (Score:3, Informative)
The publisher of the Dow industrials said that a system problem starting at 1:50 p.m. ET on Tuesday, amid unusually heavy trading volume, caused a 70-minute lag during which the value of the market measure lagged the declines in the underlying stocks.
The subsequent downward spike in the Dow occurred when the problem was corrected as the company switched to its backup system at around 3 p.m.
Just before the switch, the Dow was showing about a 160-point drop. But then the blue-chip barometer appeared to tumble some 200 points in the blink of an eye as the newly available data was correctly reflected in the average.
To err is human ... (Score:2)
Re: (Score:3, Funny)
Now, had they been running Linux and MySQL on an IBM monster box things would have probably been different
Re: (Score:2)
Now, had they been running Linux and MySQL on an IBM monster box things would have probably been different
What do you think the "backup system" was running... and how they knew the primary system wasn't working properly?
;-)
Seriously tho, I seem to remember reading at one point in time about an exchange that moved their *nix based system over to an MS based system... and kept the old system as a backup.
Don't know if this is similar.
But... we do know that they had another system running, and the two systems didn't jive... and the primary system could not keep up.
Switching over to the backup system cleared the l
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
I ran out of points ages ago, and I can still see it.
Re: (Score:2)
Capacity on mission critical systems backups where I work is identical to the primary system.
Re: (Score:2)
That isn't the WTF here...
This is the WTF. What is the point of switching to a backup, which has identical capacity, when the main is overloaded? You just end up with an overloaded backup.
-matthew
Re: (Score:2)
Many of our systems will survive the failure of a single part and fence it off, like a cpu, SAN controller cards, or memory.
However, it will degrade performance.
Sorry I was not clear on how real computers work.
Re: (Score:3, Insightful)
1. China has already unpegged from the dollar, they now peg to an index of currencies (I know, with the dollar still the main component).
2. What makes you think that's going to happen during the current administration? We both know that any negative actions will be held off on until the current "what-me-worry?" administration is