The optimal ratio between characteristics and instances in a backpropagation neural network

Question

I am trying to perform cross validation for a single use to simulate a specific problem using the Back Propagation neural network. I have 8 functions in my training data and 20 cases. I am trying to get NN to learn the function of building a prediction model. Now the problem is that the error rate in the forecast is quite high. I assume that the number of instances in training is less than the number of functions considered. Is this the conclusion right? Is there an optimal ratio for an instance?

Answer 1

(This question is often formulated in the ML literature as an acceptable size or form of a data set, given that the data set is often described as the matrix mxn, in which m is the number of rows (data points) and n is the number of columns (attributes), obviously m → n.)

In the case, I do not know the general rule for an acceptable range of functions for observation; There are several reasons for this:

• this ratio will greatly depend on the quality of the data (signal-to-noise ratio); and

• the number of functions is just one element of the complexity of the model (for example, the interaction between functions); and model complexity is the strongest determinant of the number of data instances (data points).

Thus, there are two approaches to this problem - which, since they are opposite, can be applied to the same model:

• reduce the number of functions; or

• use the statistical method to use the data that you have
  
  

Several suggestions, one for each of the two paths above:

• Eliminate "not important" functions - that is, those functions that do not contribute to the variability in your response variable. Basic Component Analysis (PCA) is a quick and reliable way to do this, although there are a number of other methods that usually fall under the heading “Downsizing”.

• Use Bootstrap Methods instead of Cross Validation. The difference in methodology seems insignificant, but the (often substantial) improvement in reducing prediction error is well documented for multilayer perceptrons (neural networks) (see, for example, Efron, B. and Tibshirani, RJ, Bootstrap Method: Validation, J. American Statistical Associations, 92, 548-560., 1997). If you are not familiar with Bootstrap methods for separating training and testing data, the general technique is similar to cross-validation, except that instead of taking subsets of the entire data set, you take subsamples. Section 7.11 Elements is a good introduction to Bootstrap methods.

The best source from this general topic that I have found is Chapter 7, Evaluating and Choosing a Model, from the excellent treatise Elements of Statistical Learning by Hasti, Tibbrani, and Friedman. This book is available for download from the homepage book.