Internet Data Mining for Investment Analysis

CaroKann writes "Reuters is reporting on a Wall Street investment research company, Majestic Research, that is using web crawling techniques to track business performance. Instead of attempting to estimate business conditions by talking to company management, or pounding the pavement visiting stores, this company uses data mining systems to collect real-time sales data and other information on companies that have a web presence. Using this data, Majestic attempts to estimate company earnings more accurately than traditional research outfits."

Traditional Wall Street Research?

But the New York-based research firm is winning converts among hedge funds who say its brand of Web-based quantitative analysis can be more accurate than traditional Wall Street research forecasts.

Possibly because "traditional Wall Street research" involves reading tea leaves and throwing down chicken bones while watching Alan Greenspan do a rain dance to the gods in hopes that our economy will pick up.

Economics and future fiscal predictions are completely theoretical. There are just too many variables involved, folks.

Now that the companies know that...

They can create bogus pages to feed to the Majestic bot like in the BMW vs. Google case.

Cue the web spam...

We can expect yet another huge rise in fake blogs, fake product reviews on Amazon and such, and paid shills in chats and message boards. Swell.

My data mining results

based on manually mining (eg reading) Slashdot I determine a spike in Majestic's share price about now...

I call Bull

TFA mentions data about drug prescriptions by hundreds of physicians. Is that lying around unorganised on the net? Tell me which algorithm you are going to use to predict how many XBOX365 are going to get sold next month by webcrawling??? You think supermarkets post their sales-figures to public webpages? Wallmart is said to have more data off-line than is available on the entire public section of the net. Now give me access to that.. But on the other hand; if you work for the sales-tax administration (in Europe) and all the big companies file their invoices weekly, that is also a good starting point...

mining online news stories for word connotations

I wrote a project in perl some years ago that would download online financial news stories and count the critical words and weigh their connotational weight, and compare that to the direction of the stock market. For example, if the words "stocks" and "down" started showing up a lot in sentences in online news stories, you might expect a downward trend.

I posted the preliminary code online in the perl newsgroup.

google "data mining" "news" "perl" etc

Realtime News Analysis

A friend of mine has developed software that goes even further. It parses streaming news stories for good/bad news and executes orders before humans even finish reading. That advantage is enough to make this company a mint.

Will it work? Yes and no.

Certainly at some level the things that happen around a company in public sources have a bearing on the stock price as many people base their investments on such info.

But the real problem with everything like this is.. even if it works well for many things... there will be those who will try to missuse it.. and finding all those will be very hard. Further it only takes one major problem case and your nice product becomes a laughing stock.

This is great

Computers should be able to give a much more unbiased assessment of the economy than any person ever could. People are essentially incapable of interpreting economic data in a straightforward way, political agendas always seem to work their way into economists opinions about the economy. By using algorithms to do the analysis (and allowing market forces to refine those algorithms), we should be able to get a much better understanding of the REAL economy.

This is a good thing for mankind.

the rise of the machines...

This is interesting stuff. I would like to learn more about the algorithms they use to analyze their data - the article has very few details. It is neat how systems like this are becoming favored over traditional human analysts (or at least reducing the need for people).

I remember back in grad school in the late 90s I worked on a major project to design an intelligent agent based system including the same functionality, but, in addition to pulling information off the internet, it could also take into account whatever other information could be gathered and interfaced into it (for example, there is also a lot of content on TV which could be fed into a system, in addition to the online data). It was a design project though and not implemented, perhaps I will need to resurrect it!

I do think the whole area of quantitative or at least semi-quantitative analysis of information, both textual and numerical, is going to explode over the next few years, driven by vast amounts of incredibly cheap computing power and bandwidth. Computer applications do amazing stuff right now, but five years from now truly "intelligent" applications will exist. The term "artificial intelligence" has fallen out of fashion, perhaps a sign of how common place these systems have now become.

As an example, our local phone company has a voice recognition system which actually works reasonably well, much, much better than anything 5-10 years ago. We are certainly making progress.

Ties to Majestic 12?

Does anyone know whether Majestic Reasearch has any connections to Majestic 12 (http://www.majestic12.co.uk/ [majestic12.co.uk])? For those who don't know, Majestic 12 is a distributed search engine. The distributed part is in that they have a bunch of people donate CPU cycles and bandwidth to run a web crawler in a SETI at home fashion. Now i thought this was a good thing to join, because we kind of need some independent alternatives to google. But if it turns out i'm sponsoring some marketing firm, well... i'd feel pretty stupid.

Will it work?

Nope, it won't, because even if it does, everyone will start using it and render it useless. There is only one trend in stock market that is backed up by statistics over long run and that is the stock market drifts upward overtime. My professor did a exmeripment using computer modeling, basically using a random number generator to decide if the stock market goes up or down, adding the 'upward drift' factor using historical data and comparing it to the actual data over last 75 yrs, and two data looks almost identical. I know it doesn't "prove" my point, but it does show that playing stock market short term is basically a flip of coin.

Several groups do this

I once interviewed with a group in San Francisco that did stuff like this. They weren't clear about who they were working for, but I do remember some of the techniques they mentioned during the interview. Some of these were actually implemented, others were just ideas:

- An eBay crawler that could estimate the number of auctions and average selling price to predict whether eBay would make their earnings target or not. eBay quickly blacklisted their IP space, so they started using a bunch of open proxies they found.

- By analyzing client/server communication for the Sims Online, they discovered that each connection was assigned a sequentially incrementing connection ID number. By looking at the rate at which the connection ID numbers were increasing each time they logged in, they determined that the Sims Online wasn't going to be nearly as popular as Electronic Arts was forecasting.

- They talked about placing a camera somewhere in Union Square (in SF) to monitor the entrace to Tiffany's during the holiday shopping season, and doing image analysis to determine what percentage of shoppers left the store with a Tiffany's bag in hand.

- Monitoring wireless carriers' spectrum to determine what percentage of GSM/CDMA channels were in use for data vs. voice. The communication itself is encrypted of course, but you can still tell whether a channel is carrying voice or data. They wanted to determine if wireless carriers forecasts about revenue from data services were accurate.

Well, then they should analyze gold

Forget trying to analize companies for optimum performance. For real performance dump the whole batch and buy precious metals because the fact is that the US economy has more debt than can ever be paid off at face value. IMHO, gold is pratically guaranteed to outperform every investmant class out there.

Seriously, just watch what happens when the fed decides to print up money to try and stall off a cascading credit collapse. They will print up some, but that will make things worse because it will drive up costs without driving up pay or driving down personal debt. So they will print up more, and that will make things more worse for the same reasons, and so on. When it is all over, costs will likely be 10 x higher while pay stas about the same. I woulnd't be supprised if the dollar stopped being a currency.

Differing Research Methods

The firm's methods differ from traditional Wall Street research, where analysts make forecasts based on conversations with company executives, advertisers, suppliers and mall visits to forecast company results and make recommendations.

This is not new

IDL have been doing this for years - http://www.investor-dynamics.com/ [investor-dynamics.com]

Maybe they'll soon announce a deal with Google?

Maybe they'll soon announce a deal with Google?

Difference between a Prediction and a Summary

Hello all:

I like to highlight that there is a difference between a Prediction and a Summary. From what I read so far, the tool posted in the article generates a summary, which maybe used as a prediction.

Let s(t) be the Summary of a system (in this case, the economy) at any given time, then:

A prediction, p(S), would be a prediction based on a set of summary S, where: S == {s(t), s(t-1), s(t-2), ... s(t-n)}

One can always make a prediction based on a very small number of summaries. |S| = 0 is a guess. |S| = 1 means that no past summaries are considered in the predication, just the most up-to-date one. Presumably, the bigger |S| is, the more information is considered in that summary.

The usefulness of such a tool lies in the value of t. Web-crawling allows one to collect much data in a small amount of time. If one is able to collect a summary quicker than everyone else, then presumably, someone using this summary tool would be able to stay ahead of the trend.

That being said, one of the input of s(t) is actually publicly available data. Financial reports events after the fact. Information based on actual financial transactions (ones that you can collect if you plan a spybot at the central booth of a major retailer, for example) is much better. At the end of the day, if you want to play a really cut-throat, high profit game of stock trading, I think you are better off having insider info.

Cheers.

B. Pascal

Real-Time?

...this company uses data mining systems to collect real-time sales data and other information on companies that have a web presence....

Can anyone name a company that keeps "real-time sales data" on the web?

The more things change...

This is a pretty common trick, and used in one form or another on Wall Street for many years.

I have seen ones that scanned EDGAR filings, (got canceled when the company was destroyed in the 9/11 attack), campaign contributions (works wonderfully for the telco and other highly regulated industries). patent filings (generally surprisingly well, though no one knows why), job adss, and many others.

I even heard of one that analyzed free internet porn...(insert your favorite joke here, but it actually was a fairly good predictor. The cognitive psychology behind it was fascinating).

Using search engines and NORA text mining is basically a form of technical investing. If you have a data store of any kind whose contents influenced or are influenced by members of the market niche of a company, it can tell you something about the future of that niche. Thats just plain marketing 101...