Advantages and disadvantages of neural networks

Question

Would I be right in saying that a neural network is good at finding “good” solutions to a problem?

I think so because they do not provide binary output for a given input, but probability, for example 0.67, may be the output.

I also think, because they are often used for generalization, they well find solutions that often solve the problem, but in some rare cases it will not.

Thanks!

Answer 1

There is no simple answer to this question. The advantages / disadvantages of neural networks is a very complex topic. Here are a few pointers:

• There is no free lunch theorem: roughly speaking, this theorem proves that there is no “ideal” machine learning method. For every problem for which a method is good, there is another problem for which the same method will fail. The methods by which he fails can be easily solved in other ways. This should always be considered when performing any machine learning.

• Neural networks are quite simple to implement (you do not need a good linear algebra algorithm, as for the examples for SVN).

• VC size for neural networks is unclear. This is very important when you want to consider how good a solution can be.

• Neural networks cannot be retrained. If you add data later, it is almost impossible to add to an existing network.

• Neural networks often display pictures similar to those shown by people. However, this is more of a concern to cognitive sciences than practical examples.

• Processing time series data in neural networks is a very complex topic.

That is all I can think of now. Perhaps others can add more.

Answer 2

This is a very broad question. In general, neural networks with one hidden layer, a nonlinear activation function, and a sufficient number of hidden neurons are able to approximate any function with arbitrary accuracy. However, the error function is not convex, and therefore the learning outcome depends on the initialization.

SVMs can also approximate any function. They are very popular because the optimization problem has a unique solution, and there may be other reasons. But recent studies have shown that neural networks, such as multilayer perceptrons, convolutional neural networks, deep neural networks, deep neural networks with many columns, etc., are more efficient and provide better results for complex applications with a huge amount of data. So this is always a compromise, as LiKao stated (no free dining theorem), and no classifier is "ideal."

Here is a document that describes the advantages of deep networks over “shallow networks” that include auxiliary vector machines: http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf

Here is a standard test and comparison of various learning algorithms: http://yann.lecun.com/exdb/mnist/

Here is a document that describes a new kind of neural network, which is especially good at solving some problems with vision (recognition of a road sign, ocr): http://arxiv.org/abs/1202.2745