Your Garbage Data Is A Gold Mine

An increasingly diverse array of geospatial, network, and time-series data is being used to generate new perspectives and insights about us.

One of the lesser understood aspects of what you can do with massive stockpiles of data is the ability to use data that would traditionally have been overlooked or in some cases even considered rubbish. This whole new category of data is known as "exhaust" data—data generated as a by-product of some other process.

Much financial market data is a result of two parties agreeing on a price for the sale of an asset. The record of the price of the sale at that instant becomes a form of exhaust data. Not that long ago, this kind of data wasn’t of much interest, except to economic historians and regulators.

A massive moment-by-moment archive of prices of shares and other securities sales prices is now key to many major banks and hedge funds as a "training ground" for their machine-learning algorithms. Their trading engines "learn" from that history and this learning now powers much of the world’s trading...