Model Generalization - Training Versus Over-training
By Bob Everly

 

Just as there are extreme values in data, there are extremes in data modeling tasks. There are 'clean' problems and there are 'very noisy' problems (explained below). There are datasets with millions of records and those with 50 records. There are those with 5 data fields and those with 500. One target output or 20. And there are data fields with skewed distributions and those with uniform distributions. The idealistic application, if it exists, would be 'clean' data with a few fields, 1 output, and with thousands of data records evenly distributed throughout the range of possible values. Does anyone remember working on such an easy task?

 

• Clean problem: model a math function with all inputs present, no measurement error.
• Noisy Problem:: model the close price of S&P500 with questionable inputs, much noise.

The above attributes affect the time, effort, and final results of our labor. Where a modeling task sets in this realm of difficulties also affects how well a model can generalize to new data. Generalization comes easier for some applications.

There are two mutually exclusive issues at play regarding model training. On one hand it is necessary that the model understand the data as well as possible, and the highest possible performance measure (like R Correlation) is desirable. On the other hand, too much learning is harmful with respect to model generalization. As a model moves from learning general patterns in data and begins to memorize particular relationships present only in the training data, the model is moving from good to bad. So we limit the models ability to learn for the sake of achieving good generalization. What if we limit the learning too much? How would we know?

One approach is to cast away all concerns about model generalization at the beginning of a project, and intentionally over-train models! Use all available data to maximize performance. While this is done un- intentionally by those new to the technology, there is merit in doing it knowingly. Performance metrics from these models provide upper-bound measurements, and we will know that they are not achievable during deployment because of generalization issues. The point being made is that it is important to first know the limit regarding model learning, before we become concerned with limiting the models ability to learn for generalization reasons.