Jacobi matrix computing for artificial neural networks

Question

I recently started thinking about implementing the Levenberg-Marquardt algorithm to study the Artificial Neural Network (ANN). The key to implementation is computing the Jacobi matrix. I spent a couple of hours studying the topic, but I cannot figure out how to figure it out accurately.

Say I have a simple forward network with 3 inputs, 4 neurons in a hidden layer and 2 outputs. Layers are fully connected. I also have a 5-row training kit.

• What exactly should be the size of the Jacobi matrix?
• What exactly should I use instead of derivatives? (Examples of formulas for the upper left and lower right angles together with some explanation would be ideal)

It really doesn't help:

What is F and x in terms of a neural network?

Answer 1

Jacobian is the matrix of all first-order partial derivatives of a vector function. In the case of a neural network, this is an N-on-W matrix, where N is the number of records in our training set, and W is the total number of parameters (weight + displacement) of our network. It can be created by taking the partial derivatives of each output with respect to each weight and obtaining the form:

Where F (xi, w) is the network function estimated for the ith input vector of the training set using the weight vector w, and wj is the jth element of the weight vector w of the network. In traditional Levenberg-Marquardt implementations, the Jacobian is approximated using finite differences. However, for neural networks it can be calculated very efficiently using the chain rule of calculus and the first derivatives of the activation functions.

Answer 2

So, in my experience with ANN and backpropagation

• The Jacobian matrix organizes all the partial derivatives into the matrix mxn, where m is the output number and n is the amount of input data. So in your case it should be 2x3

• So, let's say there is a set between 1 and k of the output number (F in your picture) and there it is 1 and I are the number of input data (x in your image), so the formula should be like this:

  Fk Jki = ---- xi 

Sorry, I don’t know how to write the formula here, but I hope my answer is clear enough. If you have any questions about my answer, please ask in the comments!