Dow Jones Plunge Fueled by Overwhelmed Computers

cloudscout writes "The Dow Jones Industrial Average dropped over 400 points today. While there were various valid financial reasons for such a decline, some of the blame is being placed on computer systems that couldn't keep up with the abnormally high volume at the New York Stock Exchange and the resulting tremor as they switched over to a backup system."

Blaming?

[VincenzoRomano]

Computers never make errors.
Humans do, at least in designing, manufacturing and sizing computer systems.
This one seems to me like blaming at a knife once you cut your fingers.

I have to wonder if this is spam related?

[erroneus]

It's a very strange and vague situation. I'm not sure we'll ever hear what actually happened. "Computer related" does tend to make one a bit frightened as to what that means exactly. It would be over-estimating the impact of stock-spam quite a bit, I should think even to consider whether or not that had any relation to the problems of yesterday. But if it did, perhaps we can now see some real action against spam... we can only hope.

Re:Blaming?

[pilgrim23]

In the 1929 crash the problem was partially blamed on the ticker tape running at times up to 1 hour late. Before computers there were people you could blame.

Chaos theory, anyone?

[pzs]

1. Computer switch-over is a bit slow

2. Market starts to waiver

3. Other parts of the market see this tremor so market waivers a lot

4. Panic ensues

5. Indices drop 10%

6. a pension company goes bust

7. my grandpa doesn't get to eat.

The last few steps are somewhat hypothetical, but still. The stock market must be one of the most immediately visible examples of chaos theory kicking humans in the nuts.

Peter

I'm actually suprised

[jhfry]

Considering the amount of, and importance of, data that flows through that system... I am surprised that it's not routinely well ahead of the needs at peak capacity.

I'd say that someone, likely the one in charge of the IT budget approval, keeps tight purse strings. Of course, he's not the one getting reamed, it's the CIO and his crew who are taking the blame even though they have repeatedly requested the funds to improve the system. Just speculation, but likely spot on.

Just another piece of ammo when I start a new job and demand a reasonable budget.

Real Cause?

[green453]

I'm all for looking at things from the tech/computer geek side of things as much as anybody on Slashdot, but isn't the summary taking things a bit far? It was mentioned that there may have been other causes that combined with computer glitches, but wouldn't the fact that markets in China dropped a whopping 9% yesterday seem to be the real cause? I'm sure swithcing computer systems may have scared a few people, but I doubt it was the primary cause of a 400pt drop. That said, it is interesting to think about the effect of computer systems on the financial markets. I've always maintained that it isn't the politicians or the business owners or the economists that run the world, it's the engineers. Think about what would happen if there was a complete shutdown of the systems that run the markets. See if all the Wall Street profiteers pay their geeks a pittance of their "annual bonus" then...

Dropping prices versus dropping data

[G4from128k]

The problem was obvious to anyone watching the markets. A trace of the Dow versus the S&P showed that the Dow's drop was NOT keeping pace with the drop in the S&P (they are normally tightly correlated, especially when big moves occur). It was clear that the NYSE's computers were woefully behind on reporting a much more orderly and steady drop. When that backup server cut in, the Dow data suddenly reflected the true state of affairs that was obvious from people watching the S&P and the broader market.

The Dow did NOT drop 200 points in minutes, the data simply caught up with the drop that had already occurred.

Don't see how they could.

[Kadin2048]

I wonder how much of this load is due to low volume day trader movement?

I would guess, virtually none, since they're by definition low-volume?

This blaming it on computers seems mostly a red herring. The markets in Asia (particularly Shanghai) tanked, and as a result, the markets in the US tanked, because companies in the US are heavily invested in China.

I think the only lesson here, in case there was anyone left who didn't get it, is that we all float or sink together. For better or worse, the US has tied itself pretty tightly to the Asian markets, and if they collapse, we're going to be seriously hurting.

"Load"? That's exactly what I was thinking

[Mateo_LeFou]

yesterday's news was annoying as hell. Everyone and their dog chimed in on what caused this horrible crash, what investors should do now, how bad it might get, etc. People: the market's been soaring for months. This is a perfect example of broadcasters' attempt to get you afraid and addicted to "news".

Re:Chaos theory, anyone?

[Lord Ender]

If an index has been trading near a certain level for a while, then a "panic" event causes a huge drop without changing the fundamental underpinnings of the market, traders view this as a HUGE signal to BUY BUY BUY, on margin if possible. In a few weeks, the index is back where it started before the panic event.

At least, that has been my observation. I can't WAIT for leveraged index ETFs... come on, ProFunds!!

Oh, and your Grandpa's pension would not go bankrupt over a panic event. That's absurd.

Of course, anyone who relies on pension companies for retirement has bigger problems...

Re:Chaos theory, anyone?

[stefanlasiewski]

0. Diversify your holdings, especially your retirement accounts, to protect against market fluctuations. The stock markets will always go up and down for a thousand different reasons-- computer glitch, bad news, hummingbirds, whatever.


1. Computer switch-over is a bit slow
2. Market starts to waiver
3. Other parts of the market see this tremor so market waivers a lot
4. Panic ensues
5. Indices drop 10%


5a. If your investments are diversified, you will survive when the Indexes drop 10%. This is especially true for long-term investments.
6. Buy low.
7. Wait for a while.
8. Sell high.
9. Profit!
10. Enjoy your retirement.

It's voodoo

[PIPBoy3000]

Yep - that sounds just about right.

Imagine a series of database transactions, with each step getting queued up and waiting for the system to finish processing it. The actual DOW number reflects fully completed transactions, but not pending transactions that might impact the outcome. This is probably a good thing, as a transaction might end up being rejected, so you only want to show the outcome of completed transactions. Once the backup system came online, the transactions quickly finished being completed, resulting in the dramatic drop.

The amazing thing to me is that the system is robust enough that transactions can survive the loss of their main computer system and bringing up a secondary one. That's database, networking, and coding voodoo, all wrapped into something pretty awe-inspiring.

Re:Blaming?

[Anonymous Coward]

In that case, the fault lies either with the person who built the computer that couldn't withstand alpha particles, or with the person who deployed a non-alpha-particle-resistant computer in a universe with alpha particles.

Re:Blaming?

[crabpeople]

"Computers never make errors."

Sure they do. Bad ram, overheated components. If computers didnt make errors, you wouldn't need CRC checking - to name but one common error identifier.

Lets try and think more, go for frosty pist less. Ok?

Re:Sooner than think.

[Red Flayer]

At that point, our economy will truly slow. Hopefully, at that time, GWB will be forced to get a real backbone and deal with China and get them to untie their money from ours.

1. China has already unpegged from the dollar, they now peg to an index of currencies (I know, with the dollar still the main component).

2. What makes you think that's going to happen during the current administration? We both know that any negative actions will be held off on until the current "what-me-worry?" administration is gone. They'll let someone else take the heat.

3. And raising prime that far will have the beneficial effect of causing inflation, which will devalue our debt. The downside will when ere high prime will cause inflation, but the returns on business investment lag behind government securities causing a shrinking economy (stagflation). We've been through it before, it's not pretty -- and it's what gave us Reagan in the White House, for better or worse.

Re:Don't see how they could.

[Lawrence_Bird]

er.. not in this case. Their 'explanation' is that they had fallen behind, which in fact was the case as when DJIA was "down" 295, futures were already down 400 and sinking. In fact, I commented to another trader what a nice arb was out there if only one could short on the downtick in the cash market. Note also that there were already some limitations in effect on automated program trading (tick rules)

In the event of a 180-POINT DECLINE in the NYA, all index-arbitrage sell orders of the S&P 500 stocks must be stabilizing for the remainder of the day. Collar will be removed if the NYA moves back to within 90 points of the previous days close.

At one point the NYA was down 400 pts. This 'blame it on the computer' or 'it was china' are BS. Stocks were over extended. While any one thing might have some effect, the time was ripe and the buyers could no longer hold the bid in the face of china sell off, weaker econ data, Greenspan, Afghanistan, Irock, etc etc simultaneously. There is too much leverage in use, too much complacency and just too much liquity. When all rush for the door in the face of trouble this is what happens.

Re:I'm actually suprised

[jhfry]

I realize that when your dealing with big iron or even high end server solutions there is little more to do than wait for the next best thing... or innovate.

Put out an RFP for someone to design a cluster based system that will perform the same functions the system currently does, but using OTS hardware in a massive single or distributed cluster configuration. Perhaps they use virtualization and have one VM for every symbol... then purchase enough hardware that only the least active symbols share machines while those with the most activity are alone on a machine. They could even be dynamically moved about as loads fluctuate.

Sure it'd be expensive... but a massive cluster of low end machines, even at $2000 a pop can still bring a lot of power and stability to the mix... and best of all, adding more power is as simple as plugging a new one in.

Anyway, it's besides the point... I just believe that someone somewhere really screwed up to let things get so bad. Perhaps it's the PM who selected the software vendor, perhaps its the CIO who hasn't done what needed to be done, or perhaps it's those with the money who didn't listen to their IT people. I personally think it's the latter case, as I have seen that attitude at all levels of business... even a 1 man company can fail to allot the proper funds to IT and let essential services go ignored (like backups).

I have begged and pleaded with bosses that it's necessary to increase capacity prematurely... but often they counter with, come back when something is broken. Then when I do, it's "What do you mean it will take 3 months to implement a fix? Didn't you see this coming?" I still believe this is what happened. Sure someone saw the problem, and knew how to fix it.. but they couldn't convince the man with the cash to cough some up and get the ball rolling. I bet the vault is open now though!