What returns `layer.get_weights ()`?

Question

I use Keras for some experiments, and I just controlled the weight update for a simple mlp model:

# model contains one input layer in the format of dense, # one hidden layer and one output layer. model=mlp() weight_origin=model.layers[0].get_weights()[0] model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) model.fit(.....) # with adam optimizer weight_updated=model.layers[0].get_weights()[0] print weight_origin-weight_updated 

For the first dense layer, I got a matrix of zeros. I thought training would not change that weight. However, the weights of the other layers vary. So I am confused, why the first layer has not changed? I checked the source code but still have not received a response, then I tried to check:

 model.layers[0].get_weights()[1] # get_weight() returns a list of weights 

This time the weights have changed. Therefore, I wonder what weight is the “true” weight that works during training? Why are there two elements in the list of weights?

---

Definition of mlp() :

 def mlp(): model=Sequential() model.add(Dense(500, input_dim=784)) model.add(Dense(503,init='normal',activation='relu')) model.add(Dense(503,init='normal',activation='relu')) model.add(Dense(10, activation='softmax')) return model 

Answer 1

For the question layer.get_weights() :

I did some tests on this and checked the source codes. I found that the Dense layer is a subclass of Layer and its weight, which is a type of python list has the weight of two elements of the layer stored in layer.get_weights()[0] and bias is stored in layer.get_weights()[1] .

It should be noted that when defining a bias layer, you can disable: model.add(Dense(503,init='normal',activation='relu', bias=False )) . In this case, the layer.get_weights() list has only one element. If you set the bias attribute to False after defining it, there will still be an element for bias , and it will be updated after fitting the model.

On the issue is not updated :

I set up a consistent model with only one dense layer:

def mlp_2(): model=Sequential() model.add(Dense(10, input_dim=784, activation='softmax', bias =False)) return model

Then I use the same method above to compile and fit it. Here is what I got:

The weight does not seem to be updated, however we can say that the weight has definitely changed. Because accuracy is increasing. I think the only explanation is that the updates on the first dense layer (which you define input_dim ) are too small for Keras to print. I did not check the more accurate value of the weights, it would be great if someone could confirm this.

Answer 2

Here is a working example.

 import numpy as np from keras.models import Sequential from keras.layers import Dense, Dropout, Activation, Flatten X_train=np.random.rand(1,10) Y_train=2*X_train input_dim = X_train.shape[1] model=Sequential() model.add(Dense(20, input_dim=10)) model.add(Dense(10, activation='softmax')) weight_origin_0=model.layers[0].get_weights()[0] weight_origin_1=model.layers[1].get_weights()[0] model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) model.fit(X_train, Y_train, batch_size=1, nb_epoch=10, verbose=1) print(weight_origin_0-model.layers[0].get_weights()[0]) #the first layer print(weight_origin_1-model.layers[1].get_weights()[0]) #the second layer 

Answer 3

There is a way to see exactly how the values ​​of all weights and offsets change over time. You can use the Keras callback method, which you can use to record weight values ​​for each training era. Using such a model, for example,

 import numpy as np model = Sequential([Dense(16, input_shape=(train_inp_s.shape[1:])), Dense(12), Dense(6), Dense(1)]) 

add ** kwarg callbacks during fitting:

 gw = GetWeights() model.fit(X, y, validation_split=0.15, epochs=10, batch_size=100, callbacks=[gw]) 

where the callback is defined as

 class GetWeights(Callback): # Keras callback which collects values of weights and biases at each epoch def __init__(self): super(GetWeights, self).__init__() self.weight_dict = {} def on_epoch_end(self, epoch, logs=None): # this function runs at the end of each epoch # loop over each layer and get weights and biases for layer_i in range(len(self.model.layers)): w = self.model.layers[layer_i].get_weights()[0] b = self.model.layers[layer_i].get_weights()[1] print('Layer %s has weights of shape %s and biases of shape %s' %( layer_i, np.shape(w), np.shape(b))) # save all weights and biases inside a dictionary if epoch == 0: # create array to hold weights and biases self.weight_dict['w_'+str(layer_i+1)] = w self.weight_dict['b_'+str(layer_i+1)] = b else: # append new weights to previously-created weights array self.weight_dict['w_'+str(layer_i+1)] = np.dstack( (self.weight_dict['w_'+str(layer_i+1)], w)) # append new weights to previously-created weights array self.weight_dict['b_'+str(layer_i+1)] = np.dstack( (self.weight_dict['b_'+str(layer_i+1)], b)) 

This callback creates a dictionary with all the weights and offsets of the layers marked with layer numbers so you can see how they change over time as you train your model. You will notice that the shape of each array and offset depends on the shape of the model layer. One array of weights and one array of offsets are saved for each layer in your model. The third axis (depth) shows their evolution over time.

Here we used 10 eras and a model with layers of 16, 12, 6 and 1 neurons:

 for key in gw.weight_dict: print(str(key) + ' shape: %s' %str(np.shape(gw.weight_dict[key]))) w_1 shape: (5, 16, 10) b_1 shape: (1, 16, 10) w_2 shape: (16, 12, 10) b_2 shape: (1, 12, 10) w_3 shape: (12, 6, 10) b_3 shape: (1, 6, 10) w_4 shape: (6, 1, 10) b_4 shape: (1, 1, 10)