What Is The Problem?

The world is full of data. After aggregation and organization, data becomes, if we are lucky, information. In today's complex and interconnected world, information increasingly exists in forms that can be stored and transmitted electronically, virtually instantly. The challenge is to truly understand, integrate, and apply information to generate and use knowledge. And a significant challenge it is. John Naisbitt's words have never been more true: We are drowning in information but starved for knowledge.

So the question becomes, how do we extract knowledge from this morass of data relationships. Clearly, there is no closed form mathematical solution. Essentially every source of data, from daily activity in financial markets, to data generated during process and product manufacturing, to data collected during medical research or patient examinations, to the billions of consumer and business purchase transactions that occur every day, is influenced in varying degrees by other data from the surrounding environment. In short, the world is a noisy and messy source of data - virtually nothing is known with certainty. Knowledge, then, is based on analysis that accommodates uncertainty. As Nietzsche said, There are no facts, only interpretations.

What Is The Solution?

Interpretation implies, in fact requires, acquiring data, cleaning data (preparing the data for analysis), analyzing data, and finally presenting data in a way that interpretations are actionable, that decisions can be made based on the knowledge gained from the data. The key is exploration and extraction - information about data relationships buried within the data itself can provide actionable knowledge. And for this, we need tools and technology to assist us. While the human brain is the most powerful pattern recognition engine we have, it's not very good at serially processing and sorting huge quantities of discrete data items. So we need to build models of the world (or activities in the world) based on data from the world - we need empirical models. In turn, models must rapidly and accurately find the patterns buried in data that reflect knowledge that is useful in the world - empirical models must learn from the data.

Why Neural Networks?

In essence, neural networks are mathematical constructs that emulate the processes people use to recognize patterns, learn tasks, and solve problems. Neural networks are usually characterized in terms of the number and types of connections between individual processing elements, called neurons, and the learning rules used when data is presented to the network. Every neuron has a transfer function, typically non-linear, that generates a single output value from all of the input values that are applied to the neuron. Every connection has a weight that is applied to the input value associated with the connection. A particular organization of neurons and connections is often referred to as a neural network architecture. The power of neural networks comes from their ability to learn from experience (that is, from historical data collected in some problem domain). A neural network learns how to identify patterns by adjusting its weights in response to data input. The learning that occurs in a neural network can be supervised or unsupervised. With supervised learning, every training sample has an associated known output value. The difference between the known output value and the neural network output value is used during training to adjust the connection weights in the network. With unsupervised learning, the neural network identifies clusters in the input data that are close to each other based on some mathematical definition of distance. In either case, after a neural network has been trained, it can be deployed within an application and used to make decisions or perform actions when new data is presented. Neural networks and allied techniques such as genetic algorithms and fuzzy logic are among the most powerful tools available for detecting and describing subtle relationships in massive amounts of seemingly unrelated data.