The fundamental difference between feedback neural networks and repeating neural networks?

Question

I often read that there are fundamental differences between root and recurrent neural networks (RNNs) due to the lack of an internal state and, therefore, short-term memory in direct transfer networks. It seemed plausible to me at first sight.

However, when studying a recurrent neural network with the Backpropagation through time algorithm, the repeating networks are converted to equivalent direct transmission networks, if I understand correctly.

This would mean that there really is no fundamental difference. Why do RNNs perform better in certain tasks (image recognition, time series forecasting, ...) than deep feed networks?

Answer 1

The fact that training is performed using some kind of trick does not change the fact that there is a fundamental difference in maintaining the state of the network that is not in the data network.

A “deployed” direct transmission network is not equivalent for the current network. This is only a Markov approximation (to a level determined by the number of "deployable" levels). Thus, you simply "simulate" a recurrent network with a k memory step , while an actual recurrent neural network has (theoretically) unlimited memory.