What factor of multiplication and additions should be used when performing adaptive training in neural networks?

Question

I am new to neural networks and, to understand this, I have implemented basic MLP, which I am currently training with backpropagation. I know that there are more complex and better ways to do this, but in Introduction to Machine Learning, they suggest that with one or two tricks, a basic gradient descent can be effective for studying real-world data. One of the tricks is adaptive learning speed.

The idea is to increase the learning speed by a constant value of a when the error becomes smaller, and reduce it by a fraction b of the learning speed when the error becomes larger. Thus, basically, the change in learning speed is determined as follows:

+(a) 

if we study in the right direction, and

 -(b * <learning rate>) 

if we ruin our learning. However, in the above book there are no recommendations on how to set these parameters. I would not expect an exact offer, since setting parameters is a whole topic in itself, but just a hint, at least an order of magnitude. Any ideas?

Thanks,
Tunnuz

Answer 1

I did not look at neural networks for a long time (10 years +), but after I saw your question, I thought that I would have a quick scout. I continued to see the same figures across the Internet with respect to the increase ( a ) and decrease ( b ) of the factor ( 1.2 and 0.5, respectively).

I was able to track these values ​​to the Martin Riedmiller and Heinrich Braun RPROP algorithm (1992). Ridmiller and Brown are very specific with respect to reasonable parameters.

See: RPROP: Fast Adaptive Learning Algorithm

Hope this helps.