IBM unveils real-time software to find trends in vast data sets

Steve Mills, left, and Dr. John Kelly of I.B.M. Research view new software that makes it possible to gather and analyze vast amounts of information at a record pace.

by Ashlee Vance, New York Times, May 20, 2009

New software from I.B.M. can suck up huge volumes of data from many sources and quickly identify correlations within it. The company says it expects the software to be useful in analyzing finance, health care and even space weather.

Bo Thidé, a scientist at the Swedish Institute of Space Physics, has been testing an early version of the software as he studies the ways in which things like gas clouds and particles cast off by the sun can disrupt communications networks on Earth. The new software, which I.B.M. calls stream processing, makes it possible for Mr. Thidé and his team of researchers to gather and analyze vast amounts of information at a record pace.

“For us, there is no chance in the world that you can think about storing data and analyzing it tomorrow,” Mr. Thidé said. “There is no tomorrow. We need a smart system that can give you hints about what is happening out there right now.”

I.B.M., based in Armonk, N.Y., spent close to six years working on the software and has just moved to start selling a product based on it called System S. The company expects it to encourage breakthroughs in fields like finance and city management by helping people better understand patterns in data.

Steven A. Mills, I.B.M.’s senior vice president for software, notes that financial companies have spent years trying to gain trading edges by sorting through various sets of information. “The challenge in that industry has not been ‘Could you collect all the data?’ but ‘Could you collect it all together and analyze it in real time?’ ” Mr. Mills said.

To that end, the new software harnesses advances in computing and networking horsepower in a fashion that analysts and customers describe as unprecedented.

Instead of creating separate large databases to track things like currency movements, stock trading patterns and housing data, the System S software can meld all of that information together. In addition, it could theoretically then layer on databases that tracked current events, like news headlines on the Internet or weather fluctuations, to try to gauge how such factors interplay with the financial data.

Most computers, of course, can digest large stores of information if given enough time. But I.B.M. has succeeded in performing very quick analyses on larger hunks of combined data than most companies are used to handling.

“It’s that combination of size and speed that had yet to be solved,” said Gordon Haff, an analyst at Illuminata, a technology industry research firm.

Conveniently for I.B.M., the System S software matured in time to match up with the company’s “Smarter Planet” campaign. I.B.M. has flooded the airwaves with commercials about using technology to run things like power grids and hospitals more efficiently.

The company suggests, for example, that a hospital could tap the System S technology to monitor not only individual patients but also entire patient databases, as well as medication and diagnostics systems. If all goes according to plan, the computing systems could alert nurses and doctors to emerging problems.

Analysts say the technology could also provide companies with a new edge as they grapple with doing business on a global scale.

“With globalization, more and more markets are heading closer to perfect competition models,” said Dan Olds, an analyst with Gabriel Consulting. “This means that companies have to get smarter about how they use their data and find previously unseen opportunities.”

Buying such an advantage from I.B.M. has its price. The company will charge at least hundreds of thousands of dollars for the software, Mr. Mills said.

Link to article: http://www.nytimes.com/2009/05/21/technology/business-computing/21stream.html