Comment Hopefully a return to real science (Score 4, Insightful) 87
The problem with 'Big Data' is everyone is trying to use it as a substitute for actual hypothesizing and experimentation.
I am not suggesting it isn't useful, it is, and it can be a huge help in identifying non-intuitive relationships that may exist. Its not being marketed that way though! Everyone is trying to sell it as the solution to all their unresolved problems and knowledge gaps.
At the end of the day all it can ever show is correlation, never causation. All the fancy AIs we add on top are really just correlation engines as well. One day real-soon-now WATSON or something like it will diagnose your cancer. It won't 'discover' the cure though, it will just apply the 'KNOWN' treatment that statistically correlates with the best outcome, hopefully excluding some which correlate with especially un pleasant side effects.
Same is true with the financial markets. Big Data alone will never discover a unified theory that explains market behavior. It will probably make a handful of people stupid amounts of money based again or event correlation and speed. As long as those are the drivers though we will remain forever at risk of sudden meltdowns.